It's not set in stone, its just used as an excuse. Or a thought terminating cliche.
From OP:
> the First Amendment (with an exception existing for pornography)
The text is not
> Congress shall make no law abridging the freedom of speech, except for showing boobs on TV, which would be a heinous crime and worthy of deportation
There are many exceptions to "freedom of speech".