People aren't allowed to drive drunk not because it's a risk to themselves, but because it's a risk to other people. This is the same with the vaccine. You are putting at risk children, the immunocompromised, or people who have legitimate medical reasons they can't get the vaccine, do your logic falls apart. Not to mention the risks and costs associated with an overburdened healthcare system.
Actually, refusing the vaccine isnt the same as driving drunk. To be in a situation where you're endangering others by driving drunk requires an action: drinking alcohol recklessly (action) Refusing the vaccine requires no action on the person's part, but somehow are still considered to be endangering others (even if they're healthy and without COVID).