Fame doesn't affect all accusations the same way. I believe the probability of false accusations raise faster with fame than the probability of true accusation (which is mostly dependent on actual guilt).

There's even the possibility that the probability of true accusations lowers as fame raises, leaving more room for the false ones.