It's not unambiguously true that freedom-of-speech itself is substantially responsible for anyone's loss of other rights. We can't even have a public conversation about this without freedom-of-speech, where all relevant ideas and evidence can be openly evaluated, unless there is essentially no gatekeeper (committee, policy, "filter", etc.) deciding on what is relevant or permissible speech. We will only have biased answers if we bias the discussion.

Further, even mere perception about the bias in the discussion (caused by censorship etc.) causes some people to disengage from the discussion altogether, and facilitates the "silo-ing" of groups of people who have little contact with each other. That is not a recipe for a functioning pluralistic society, and ultimately for peace itself.