How would we ever know if men actually were worse than women at math? There will always be a sufficient number of confounding variables as to have plausible deniability, not that it really matters. I think the line for when we're grasping at straws (or when we have enough evidence to draw a conclusion) is different for different people, which is why there is debate over such stupid things in the first place.