Yeah we find this type of optimization all over nature. Even radiation is important to create noise. We need it in machine learning. A noisy optimizer is critical for generalized learning. Too much noise and you learn nothing but no noise and you only memorize. So there's a balance

I get the "outliers are useful" thing you're trying to emphasis. But as someone from a mountainous country, please dont "climb[ing] mountains in Crocs", we regularily get media reports of hopelessly underequipped people having to be rescued with a whole team of people, in the middle of the night, with horrible weather, usually also endangering the people that do the rescue. I guess what I am trying to say is, there is a limit to how silly you can/should be.