I'm not sure that's entirely true in the UK. The Polish plumber taking British jobs is a fairly common trope in far-right discourse. I believe this is prevalent across Western Europe in general.

I would think that the majority of immigrants that are not white are also not from the EU, but I don't know that statistics.

That is not true.

The leaders of both the Brexit campaigns (Boris Johnson and Nigel Farage) both clearly said that they wanted more non EU (so mostly non-white) immigration - provided it was skilled people. Government policy since Brexit has made non-EU immigration easier.

Remainers wanted less non-EU immgration and more EU immigration.

So somehow the people who wanted less white immigration and more non-white immigration are the racists?

This is one reason a lot of us Brown people voted for Brexit. not my main reason, which was mostly opposition to further integration (the commitment to "ever closer integration") and some aspects of EU decision making, legislation and regulation.

Kinda. I think it's slowly becoming more anti immigration in general.

Though with the London Mayoral election on Thursday, it seems like people want Khan out, using "ULEZ" as the excuse for not wanting a "brown" person. I know a fair few people who live in London and their only criticism of him is ULEZ, even if it doesn't effect them at all (massively brainwashed by Facebook)