Restricted immigration that allows the top talent of various countries is not the same as mass immigration that replaces the host population and culture. Your dream of an open-borders utopia is incompatible with a successful country.
Israel has had a strict immigration policy, far more restrictive than the post-1965 US policy. They don't seem perturbed by your claimed economic requirement of not "exclud[ing] everyone who wasn't smart enough to be born there" (while you've already shown you really mean open borders).
The US prospered with very restricted immigration for decades.
> Keep them out, and they'll concentrate in other places.
Let them concentrate, especially those from the Third World, so their countries have a chance to benefit from their talent and develop. I don't understand why you are against their home countries prospering.
Xenophobia-at-a-distance has no place in modern civilization.