The tyranny of the majority is not an anarchistic idea and yet is exactly what is being invoked when the argument is made that it's acceptable to ignore the concerns of a minority population because doing otherwise would be unfair to the majority.

I think this quote from Tocqueville explains it well: "So what is a majority taken as a whole, if not an individual who has opinions and, most often, interests contrary to another individual called the minority. Now, if you admit that an individual vested with omnipotence can abuse it against his adversaries, why would you not admit the same thing for the majority? Have men, by gathering together, changed character? By becoming stronger, have they become more patient in the face of obstacles?"

What matters is not how many supporters each side has but what the merits and demerits of each side are.