Rishi Sunak, our current prime minister, is brown, and I don't think that's really come into play at all.
Sadiq Khan is Muslim which is more of a wedge issue, but I would say in my circles, ULEZ, and more generally anti car sentiment, is a huge concern.
In my experience as a Brit no-one really cares about skin colour but about culture, religion (if fundamentalist), accent, etc. It basically just comes down to "are you integrated". I don't think that it's unfair to expect people to fit into society.
That's pretty much what the immigration debate is all about. If a Nigerian millionare comes over, brings his family, whacks them in a private school, basically no-one cares. Bring more.
It's unskilled, uneducated people who have issues with integration that basically everyone wants to limit.