It doesn't have to be force at the start. If a politician were to run on the platform "no free speech for poor people", obviously they're not applying force or threatening it.

And even when elected, they threaten no force, they simply remove a right from the law.

At no point did they threaten force against poor people, only that their free speech be taken away. They got elected and are doing it non-violently.

Now poor people no longer have free speech.

Hence, either you must treat a threat against free speech as violence unconditionally of it's physical or psychological component (ie if it has a legal component) or you must acknowledge that free speech can be threatened by non-violent oppression.