That's why I mention biases, which is a concern as AI becomes more and more ubiquitous. This is admitedly a silly test, I don't mean to dismiss the whole project because of a single response. I just find it interesting that, because most humans would be tricked, AI tools based on human generated data are inheriting their biases (conscious or unconscious).

Imagine if (or when) these tools were used to make more serious decisions, like hiring or sentencing:

For example, if an hiring AI disregards a female candidate over a male candidate with the same experience for an attorney role because statistically the male candidates fits the role more even if resumes are otherwise similar.

Or a sentencing AI infering crime is more likely to be committed by some groups, purely because those groups are currently over-represented in the prison population...

I think what's more "unfair" about this is that there actually is information which implies that it's the paralegal who is pregnant. The "X married Y because she was pregnant" scenario is more likely when Y is going to be put in a particularly bad way because of the scenario, and X can reasonably take care of her. It's also more likely to occur when there's a power imbalance, where X is more powerful than Y, and therefore feels responsible to "amend for" the situation.

"Male attorney gets female paralegal pregnant" matches both of those templates pretty well, and so "...and so does the responsible thing and marries her" fits. "Male paralegal gets female attorney pregnant", not so much: The power / provision dynamic there is completely different, and so "...and does the responsible thing and marries her" doesn't really follow. If they end up getting married, it's because the more powerful and more highly-paid attorney decided that's what she wanted to do, not because she was making the best of a bad situation.