There probably is a very general consistent rule to your/your source's idea of free speech, like "everybody's liberties should be curtailed only where they endanger the liberties of others". But implementing this leaves enormous scope for arbitrary and creative judgments. We can assume that your opponents, being wrong, have less internally consistent ideas, but even so the presence of exceptions in itself is nothing to be critical about if your own conception of free speech is nuanced. They may be following some logic too.