I think we're in severe danger of spiralling toward epistemology here, but there's a huge difference between "nobody knows" in the sense you're using it and "nobody has any idea". Society (or more broadly biology) doesn't need us to get it right 100% of the time; these are very, very rare conditions. Going off secondary sexual characteristics is going to get you the right answer as near as dammit every time, especially in premodern contexts, and society has always been happy to work with tiny or even not-so-tiny uncertainties. (Is that kid really mine? Is the accused really guilty?)

The way I see it, the sex binary is fundamentally about reproduction. It's why we can use the same concept for everything from pondweed to platypuses. All across nature, male=small gametes, female=large gametes. In humans that's driven (with the potential for things to go haywire occasionally, sure, but still driven) by the XX/XY system, so that strikes me as a reasonable thing to base a definition on.

Side note re "nobody knows whether anyone is a man or a woman until they get tested" - I'd say that giving birth or fathering a child is a pretty big clue. AFAIK the only cases where that doesn't line up perfectly with genetic sex relate to mosaicism, where I'll freely admit my intuition goes completely kablooie.