The character flaw isn't worrying about the labels being foisted wrongly upon someone. The character flaw is someone worrying that their personal politics can't clearly be distinguished as not being motivated by hate. Odds are that is a marker of a bad and unjust political opinion.