I feel like voting for the content itself but keeping the comments section flat and sorted by time would lead to less gamified and much better comments sections than what we have now.

Maybe keep upvotes around as a positive reinforcement without having an effect on the position of a comment, and obviously mod flagging for things that break rules, but upvoting and downvoting being equivalent to "I agree" and "I disagree" leads to users gaming the system to hide things they disagree with.