(Ref: why were medieval cities so dense despite their low populations and plentiful surrounding acreage? Everything had to be walking distance, so a big manor in the middle of the city would place a major burden on everyone's commute whose destination was not that manor).

So the tax is what you pay for the burden that your exclusive hold on the land imposes on others.

Or, you could think of it as what you pay to maintain the right for it not to be paved over as a superhighway, plus what you pay for the government to agree to keep other people from trespassing.

Not the fee for living there -- the fee for not letting others live there.