Because it's the best (probably only) way to prevent every actual Reasonable Position from overriding every other factor and consideration. If freedom of speech is the default, there's no way an extremist group will stop conversation about any particular issue.
> Thus far people don't seem to be applying the badness of Nazis to non-Nazis (at least not intentionally)
This is Godwin's Law, which is in itself a testament to how common this is.