There's a difference between displacing the local population (which you imply) and moving somewhere there's better opportunities (jobs, housing, &etc). As long as there's no coercion or violence involved then who's the victim if my new neighbor is from Guatemala?
Plus it'd be pretty hard to be an oppressive dictatorship (looking at you, California) if the people could just up and move to a less oppressive regime (like Arizona). Of course I'm (mostly) joking with my example but a CA->AZ mass migration is happening as we speak and nobody is calling for a wall on the border (yet). They can't put up massive apartment/condo monstrosities fast enough to handle all the new folks moving here and presumably they're finding jobs without too much trouble.
So, yeah, I think if governments had to compete to retain their "subjects" the world would be a much better place and people probably wouldn't need to relocate unless they really wanted to unlike today's multiple "migration crises" (their term, not mine).
