It's all relative. If everyone else moving in can spend a mill plus, you're priced out. If you and your neighbors happen to be the only ones improving things, and the mil plus group comes in and stagnates/displaces you through gentrificative forces, requiring you to to start from scratch elsewhere.
Lets say you live long enough for the cycle to repeat multiple times. I assume proponents would say, "well, hooray, everyone but you is way better off, I guess it sucks that you weren't better at managing money", when all anyone ever wanted to do was live together and be left alone.