That "retirement, mortgage and rainy day fund" could very reasonably be more than $1M. Especially as the $1M figure here includes your home, which isn't something you can extract an income stream from without grave risk.
(Back of envelope: Suppose you reckon you can safely spend 5% of your initial retirement fund per year, without running out before you die. And suppose you want $50k for medical emergencies, because you're in the USA and aren't sure Medicare will (1) still exist and (2) cover everything. And suppose you want $30k/year of effective income after retirement, which doesn't seem insane. Then you need $650k in that fund. Note that that 5%/year figure depends on the overall growth of the economy and how long you expect to live after retirement.)
Yes, saving a lot of money can just be pointless avarice. But I think the level at which that becomes a reasonable suspicion is rather higher than "total assets of $1M".
