Once upon a time, neighbors also collectively agreed to zone by race and ethnicity, until Buchanan v. Warley in 1917 put an end to that practice and created the minimum-lot-size SFZ proxy system we have today. You can argue persuasively for respecting SFZ, but you probably can't do it from a moral high horse (which is just to say: you can't rest your argument on a moral right).