Aside from that, there's an important difference between expressing an opinion, and creating an atmosphere and culture of hate and/or violence.

There are things you're not allowed to say because they endanger lives, like threatening people, blackmailing people, or slandering people. Spreading intolerance and hatred directed at specific demographic groups has also been shown to endanger those people, so banning that is absolutely defensible.