Ethical non monogamy requires ethical behavior. If you’re seeking out students wives, that is predatory and unethical behavior. If you’re also denigrating women that is unethical behavior. While it’s possible that Feynman, the students, the students wives, and Feynman’s wife all consented to this behavior, it seems unlikely.
I do not believe “that is on the wife to say no, not on him”. If these allegations are true, he was a teacher engaging in predatory behavior. That is simply unethical due to the power imbalance. And the fact that he was asking people to cheat is unethical.