You can't control what other opinions people will have of you, or suppress their expression of those opinions, that's true. But that itself is part and parcel of freedom of speech. The social consequences of expressing opinions that other people find offensive or disgusting are yours to bear, and may indeed motivate others to want to have little to do with you.

But this is juxtaposed against an expectation of freedom of speech as a cultural norm -- the point above is precisely that the way this is expressed in other contexts is different from the way it's expressed with respect to the political state. So we do have an expectation that formal institutions will not engage in prior restraint, or punish people for speech itself within most (but not necessarily all) categories, etc.

For example, people might find it reasonable that someone be dismissed from a job for making overtly racist comments in the office. But the same people would consider it inappropriate if, for example, an employee was fired for expressing that he does not like the boss's favorite movie.

There are absolutely different boundaries and different weighing of consequences between how we apply the norm of freedom of speech to political vs. social situations, but to say that we do not adhere to freedom of speech as a cultural norm at all is quite incorrect.