The "first" kind of comments are easy to detect, so I think they wouldn't be a big problem
The chances that a post written later in the game will become one of the most voted because it's really a good comment are close to none anyway
Another interesting feature would be highlight and prioritize content written by users you follow, based on some simple rule like how many votes you gave them before or how many interactions you already had in the past
But downvotes are a prize and a status, you have to reach a certain amount of karma points to be able to downvote something so I guess unfortunately they are here to stay
With this kind of system, you could even boost downvoted comments as long as they also receive upvotes, encouraging more diverse discourse. If we want an online space that doesn't become an echo chamber, we need to make it okay to respectfully disagree.
[0]: https://pol.is/home - pol.is uses votes to find common ground between divisive groups
Congratulations, you invented Usenet. :)
Back in the day we didn't have voting, we had kill filters. If you didn't like another user, you'd have your reader filter out their messages. There were far fewer kids on my lawn in those days as well.