I think one crucial difference between buying a house and an index fund is people are usually highly leveraged when they buy a house.
Example.
Assume you
1. Buy house and put down 100k for 500k house.
2. Buy 100k of stock.
3. Both assets drop 20%
Your stock will have lost 20% of it's value and you'll have lost 20k and 20% of your investment in the index fund.
Your house will have lost 20% of its value and now at 400k, you've lost 100k of net-worth and 100% of your investment in your house.