I do generally agree with you that the accredited investor definition should just be scrapped entirely, but the problem with fraud is that you usually don't find out about it until your investment is gone, and prosecutions often aren't able to recover anywhere near enough to make the investors whole again. And even when they do, it can take months or years.
Someone with a $20M net worth can deal with losing $1M. It sucks, but it's not going to put them on the streets. Someone who puts all of their savings into a fraudulent investment then becomes one car break-down away from not paying their bills. Consider also if that person is retired and is living on a fixed income.
The kind of people in the latter group are probably more likely to fall for a scam or fraud. I would love to lift investment restrictions -- I absolutely agree that this is a "richer get richer, poor get poorer" issue -- but not without a way to better protect more vulnerable investors.
The first thing that comes to mind is some sort of government-provided investment fraud insurance. But then I worry about perverse incentives: every time someone's investment tanks, they're incentivized to try to prove that the investment was a fraud.
