I don't think it's about labeling people as being good or bad, but more about not letting things stand.
If you encounter attitudes from people that are hostile towards you, you can either speak up and allow them to correct themselves (or double-down) or be silent and give your tacit approval.
I think sometimes opinions are best left to change slowly or through exposure, when you approach things in this way it often just comes off as hostile and that doubling down you're talking about becomes a much more common response.
