People want single family homes in neighborhoods of single family homes, though; the value is highly contingent on the neighborhood those homes are in.
You might be fine with a given backyard in a suburb, but not with a highrise next door that can see right into it.
Zoning handles this in the same way that it handles preferences about people not wanting to live next to industrial shops or giant supercenters. It does this by restricting what can be developed, even if someone moving away doesn't care what happens to their old lot, and could make more money on the sale otherwise.
Whether we should respect those preferences is a fine question, but zoning is pretty much the only tool to enforce these kinds of commons-oriented preferences.
