I don't disagree, but my point was that they shouldn't. Clarence Thomas famously wrote about believing that his own success was tainted by affirmative action, and later he was indeed appointed to the Supreme Court primarily because he was a black conservative. That doesn't mean he was unqualified or less fit for the court than previous appointees, no matter what his own insecurities were, and it also doesn't mean that he or our society would have somehow been better served by Clarence Thomas not being given any of the opportunities affirmative action programs may have afforded him.

The deeper problem is that there isn't really a solution here. The old system wasn't neutral and instead actively discriminated against huge swathes of the population. That the beneficiaries of that system never doubted their worthiness doesn't make the old system better, and any change that impacts them will suffer from the same aspersions about a lack of "merit" that we are seeing now.