Because it's often spread over many years of a mortgage, I can see why SOME people might not. It is not as concrete as someone stealing your car, but the amount is in the same ballpark
But some people should care - these are the same people who track their stock portfolios closely, have college funds for their kids, etc.
A mortgage is the biggest expense for many people, and generally speaking I've found that people don't like to get ripped off :-)