But as tax money is raised, the local government has a choice

1. They can use the tax money to fund services.

2. They can return the tax money collected as a Citizen's Dividend.

If they don't have any need for any more services, everyone would still be able to live there. Your concern that people will lose their community are unfounded. They might decide, as the tax goes up, that they don't need a 1 mile by 1 mile plot of land, in which case they can sub-divide it and live in the area they care about. Anyone who uses less than the average amount of land ends up net positive through this process. Since land ownership tends to be skewed by wealthy land-owners, this means the majority of people receive net proceeds.

But maybe the city does want to spend these new funds on roads and infrastructure for the new people. They have correctly identified that land values will raise by more than they spend, so it is in the interests of the community for them to build. Since these services are profitable to the community, the collected dividends will now be even higher. This represents even more gains for our original garden tenders, but everyone who joined after the initial process shares in these gains.

Now that they have built more awesome stuff, the original citizens are now much better off. This city is now filled with companies moving in, providing jobs for the citizens, it's a magical process. Land values are all much higher. This means that the tax is higher, but it means that the dividend is higher. They can now either go for #1 or #2 again...

In a non-LVT world, it wouldn't make sense for the infrastructure to be built for the sole profit of the original land owners, so it wouldn't be built. Everyone is better off with the land value tax.