If there are not enough chargers, won't that create an incentive to build more? And yeah, you probably can't build them fast enough, or the grid capacity will not be sufficient, but won't people then just move to places where they can? Or switch to alternative modes of transportation?

I think the fallacy is that people just want to replace ICE cars with EVs as they are, and then realize you need a lot of space and grid capacity to charge them, etc. But because of the scaling problems you mention, there will be a very large pressure to build better public transport. If people can't charge their EV, or can't afford a car, or don't have good transportation, they will move somewhere else. This is structural change, and yeah it is going to be inconvenient and expensive (and we could have done it better with more foresight), but ultimatively we will figure it out.