Firstly, I don't think anyone ever thinks that those traits can only be found in men. It's just that they believe that its more likely men to have those traits. Like the chopstick example I gave, no one is ever going to say only asians use chopsticks, and if you told them that the person in question was actually white, they likely wouldn't bat an eye. But their first guess would be asian. So it's not really a problem, because that situation doesn't really exist.

But say we adjust and say the problem is that people think its more common for men to have those traits. Then:

A) Someone comes up with a very good way of measuring those traits. Say it actually turns out to be pretty objective and repeatable and consistent. And then they go out into the field and test a representative sample of North Americans. And then they find that amongst North American men and women, men do score higher in those traits by meaningful measurements of "higher" (say both a higher mean, and an asymmetric distribution shifted towards the higher end).

Now, I'm not saying that that's true, nor do I want that to be true. But if it -is- true, then it turns out people are right! Their intuition matches reality. Now what? Maybe the problem is that its actually true, and we have to dwelve into nature vs nurture and all that "good" stuff. And this is honestly something we have to consider. Given the self-reinforcing nature of society/cultural pressures, and the non-trivial possibility that there are actual biological differences that will bias the traits of the genders, it's actually possible that some sort of difference could be found.

B) Same thing as above regarding reliable tests, but the tests come out negative. No meaningful differences between genders. Well then yes, we have a problem.