I'm not saying people should stay in places where they are ostracized. That's not healthy. But I think it's also a mistake to withdraw from places and activities before we've even had a chance to become ostracized.
An example: I'm a gay atheist from Idaho. I have extended family members that look like they belong on Duck Dynasty. Each family event, me and my husband are presented with a choice: we can skip the event and its awkwardness, or we can join the event and face it head on.
Each time we attend these events, we leave with the same impression. "That wasn't so bad," and from my husband, "Your family is actually super nice." And because of this interaction, they become less homophobic, and I grow to understand redneck values a bit better.
Besides, I've learned over time that what I think they're thinking about me is actually much worse than what they're actually thinking about me.
But each time I'm invited to one of these events, my first gut instinct is not to go, because it's work, and it can be awkward, and it's much easier for me to spend time around people who are more like me.