I disagree that “nothing can be done”.

There’s lots that could be done that don’t fall in to your description of affirmative action.

Invitations could be extended to people to submit proposals for a talk. This already happens, but maybe the circle gets expanded once it’s evident that the circle has been too small. The simple act of asking is often enough to make people feel welcome.

I also call BS on the claim that you’ve never discriminated against someone. No person is a saint like that. Every human and sentient creature has inherent biases.

Maybe you’ve never actively discriminated, but you most surely have passively done so at some time in your life. For anyone to claim they are perfect in this regard is only fooling one’s self.

That you then go on to say that it’s not there 95% of the time withoit any actual reflection on the matter perhaps shows that you’re not well versed enough in the dynamics of race and gender relations to make that call.