In substance I agree with you. But at the same time I think HN would be better off without downvoting at all. An ethos of free speech is entirely lacking.
I don't know why you would expect an ethos of free speech from a privately-owned, privately funded service provided to you at-will by venture capital. YC isn't a charity organization.
On the contrary: we know, from the paradox of tolerance, that some speech must not be tolerated. That's philosophy, but real world experience backs it up. The parable of the Nazi bar has roots in the world, and we have the example of the heat death of Usenet as well