I agree that the broad reading of the interstate commerce clause seems dubious, but the same precedent [1] underlies the controlled substance act, clean water act, violence against women act, endangered species act and a whole lot of other laws. Unless you are willing to hack away a large chunk of US law, you can't really undo that one.

[1] Wickard v. Filburn ruled that growing wheat FOR USE ON YOUR OWN FARM can be regulated as interstate commerce because otherwise that demand would tend to raise prices on a interstate grain market.