I don't think that is a concern. I had this exact conversation on Reddit a couple days ago. What I wrote:
>The flip side is it's much easier to make up the difference due to the increased economic opportunity in <high cost of living area>. If you can't it's a good incentive to move somewhere cheaper in turn reducing the pressure on housing in <high cost of living area>.
Say I'd like to live in Monaco or Luxembourg. No one owes me that privilege. I can't just demand to get that. Even if I get a job in a grocery store or whatever to legitimise my residency.