I'm playing along with your analogy here, but I actually think it's a terribly weak analogy. Racial discrimination is often so ingrained in the culture that its perpetrators aren't always hulking bullies menacingly taking lunch money from people. They are more dangerous because they often see themselves as kind, empathetic and intelligent - often believing that they can detect and solve deeply complex, emotion-rooted problems such as racial discrimination simply by thinking about it for a bit when it's convenient for them. They can't fathom that the experience of the discriminated may be so vastly different than their imagination permits, simply because they haven't experienced it firsthand.
Too many of us have been silenced and told to shut up both explicitly and implicitly. When it comes to racial discrimination, we will not go unheard. In this instance, in this matter, our voices do weigh much more than yours and that's simply the fact you'll have to live with.