At least around here (Boston area) there are really three things of value with a lot:
1. The land value.
2. The structure value.
3. The right to have this structure on the lot.
That is, if you tore down the structure you wouldn't generally be able to build anything anywhere near as large, because the building predates zoning.
Assessment ignores (3), and groups it in with (2), which then depreciates. Which isn't right, because (3) doesn't depreciate