It's not a position of "someone is likely to do me wrong", it's a position of "someone is likely to do society wrong". Interacting with someone isn't an accusation...
It's a well supported position by the fact that I seem to witness theft (bikes, from cars, etc) in plain view on the street with a frequency of about once a year. Double that if you include things like hit and runs on parked cars that aren't exactly malicious but more crimes of opportunity.
I suspect that frequency is higher for me than for most people - simply because I walk a lot and I live downtown in a big city (Toronto). But if you change "6 monts to a year" to "3 - 6 years", applied over a city of a few million people that is still extremely significant.
I suspect things like police abuse of power are much rarer still, but the impact from them is so larger it's also a significant factor (why there is the huge movement for bodycams...).