The issue is that the behavior of this other group isn't benign. It comes with a very real "comply or else", which already affects my ability to participate in society due to a particular transient medical choice. It's a tough sell that medical decisions are best made under threats of harm (legal, livelihood, societal participation, etc..).

I think it's fairly natural when someone tries to coerce you into something to reject it on principle alone. There's big difference from "you should stop eating so much McDonalds because of downstream effects on the medical system" and "you WILL stop eating McDonalds or we will take away your autonomy to do so." I will not be forced into medical decisions by a mob threatening harm. If/when I get the vaccine, it will because my view of the risks/need ticked in favor of it (in theory anyways).