I don't think it's so much that you can't tell the difference between crazy bad and crazy good ideas in the early stages, it's that you're asking the wrong people. pg loves to quote the mythos that AirBnB was such a "crazy" idea early on that big VCs couldn't see the potential. But even in the early years, I don't think AirBnB was a big leap at all. Couchsurfing was already big, and platforms like HomeAway/VRBO had received huge funding rounds. pg was just asking the wrong VCs - not surprising that a bunch of rich old guys would be put off staying in someone's guestroom. Indeed, my understanding is that the first investor to really see value in AirBnB was very familiar with the vacation rental space.
In my opinion, the startups that were wildly successful weren't so much because they had "out there" ideas, but because their execution was unparalleled. Dropbox, Stripe, Figma, AirBnB, etc. just came out with products that were a delight to use when you first tried them in ways their competitors weren't.
Trying to be introspective, I think the event that most surprised me and seemed "crazy" was the election of Donald Trump in 2016. But again, I think i was just asking the wrong people. My circle of college-educated urbanites was basically unaware of the depth of some of the discontent with the status quo that Trump tapped into.