Only ~5% of homes are second homes, not 33% as in your example.

Now suppose there are 25 homes and 30 families, better reflecting current reality. If you can't make more you're screwed. But you can create new housing. Zoning makes this expensive, because zoning requires it to be done in an expensive way. You have to destroy a large building to build an even larger one because multi-unit buildings aren't allowed where there are currently single-family homes. But you can do it. It just costs $600,000/unit as a result.

Since there are more families than existing homes, everybody bids up the price of housing units because they don't want to be homeless. When the prices hit $600,000/unit, construction companies build more units because now it's finally profitable. Then all the units stay that expensive because there are only just enough -- or not enough, but the remaining people don't have $600,000. But they don't go higher than that because that's the price at which more can be created under existing zoning rules.

If Richie Rich comes along and buys two second homes, that doesn't raise the long-term price, because the price is right at the threshold of profitability for new construction. So two new units get built at that price and the market price stays where it is -- at the cost of creating more units. If you want units to cost less, that cost has to go down, because units can't be added for less than that and eliminating 100% of second homes is not enough of a reduction in demand to shift from that being the price-setting factor.

Whereas if you could build new units for $200,000 then Richie Rich could buy a hundred of them and they'd still cost $200,000 because construction companies would just build a hundred more.