I'm not crossing a street with you if you're carrying $500K in your backpack everywhere you go.
Physical possession of wealth is a bad long term strategy. Eventually people WILL find out, and you WILL become a target.
One of the main functions of government is private wealth protection. Banks are a feature, not a bug.