You wouldn't, but they key is that the cyclist/pedestrian doesn't
know it's you.
Say it's 5% of drivers who are maniacs, one encounters many many cars, and the cost of misjudging this situation is "grievous injury". So the end result is people will give way to cars even when they don't have to.
> Not every guy is big and strong and capable of or wanting to do violence.
This is structurally comparable to the way women treat "all men" as "potentially violent". It's not about you per-se, just a consequence of the group you cannot be immediately separated from.
Speaking of, that's the other half. Sure you're a normal guy. You don't want an assault change. The tradeoff changes when you are at risk, when violence is being done to you. (Hence the harassment-of-women example)
Sure you're not gonna try to murder the person outside like the road-raging maniacs, but when your safety is on the line, driving dangerously close past them is on the table.