It would "penalize" those people if they owned farmland in areas where there is high demand for housing (and the land itself is valuable). The whole point is to tax all land at the value people would pay just for the land - and as a result would shift activities like farming and logging into less populated areas.

It's a fair concern, but also because property tax already includes some land tax it's not dramatically different. Those activities are already mostly done in areas where land is extremely cheap.