Someone with a $20M net worth can deal with losing $1M. It sucks, but it's not going to put them on the streets. Someone who puts all of their savings into a fraudulent investment then becomes one car break-down away from not paying their bills. Consider also if that person is retired and is living on a fixed income.
The kind of people in the latter group are probably more likely to fall for a scam or fraud. I would love to lift investment restrictions -- I absolutely agree that this is a "richer get richer, poor get poorer" issue -- but not without a way to better protect more vulnerable investors.
The first thing that comes to mind is some sort of government-provided investment fraud insurance. But then I worry about perverse incentives: every time someone's investment tanks, they're incentivized to try to prove that the investment was a fraud.