The problems are that it becomes visible and hence it generates fear of the unknown which some politicians (populists or demagogues is probably a better description) easily exploit.
That really is the core of the issue - and it’s a big one, not easy to solve.