Anyone know anything about wasps/hornets/black stinging insects...?

So, the screen in my bedroom window is broken, and doesn't fit the window frame tightly. These past few days, I've noticed what appears to be a colony of wasps just chillin between the screen and windowpane. They just hang out in groups of 5 or so. They come and go like they own the place. I'm assuming they've built a nest...somewhere. Maybe in the wall. They don't seem to be able to get inside--at least, I've not found any in my room. But does anyone have any past experience with this sort of thing? Should I be concerned? Or should I just sorta let them be? They're reddish-black, about an inch long, and have mean looking stingers. My google-fu tells me that they may be the relatively gentle giant black wasp, but they may also be a type of hornet, which could swarm violently at things like loud noises or vibrations (better put away the sub woofer, i guess...) What would you do in this situation? (I should add that I may be allergic to them. I've never been stung by...anything, so I don't really know. Yes, I've really made it this far in life without EVER being stung by a bee.) Any advice would be welcome.