The idea behind the LVT is that the tax is based on the best and highest use of the land. But zoning is a component.
If the market is begging for a 100-floor tower on a piece of land, but the zoning says you can only build a single house, the LVT would be based on a house sitting there.
So, if you had a historic or otherwise significant structure, you’re going to have to downzone the land to prevent the loss of the structure. Naturally, that’s going to be abused. Connected people are going to get their property zoned for smaller structures, while unconnected people will get their property zoned for huge structures.