That's already how civilization works. There's no one person that knows everything about (say) modern food production, from top to bottom. If it ever stopped working (because too much knowledge was lost somehow), most people would die. And yet the system seems fairly resilient. Mostly, only local knowledge ever seems to be necessary to keep the whole thing running. Super-intelligence (or even just super-normal-intelligence) might expand the scope of what constitutes local knowledge but it will still run into limits somewhere.

Noble "save the humankind using the tech nobody fucking understands" textbook goals like curing cancer, solving global warming and achieving interstellar travel would always turn up when owners of trillions of dollars place orders on positive AI narratives, but in reality all that will wither down to "It's what plants crave! It's got electrolytes."

I think anything is and will be explainable. Like in the OpenAI proof, I’m sure they were able to understand the solution 100% and could even drill down and ask more clarifying questions to the model. After all, the point of science is so that knowledge can be made logically transparent. If something can’t be explained, it isn’t really understood yet — and the same applies to model outputs. The only question is how much effort it takes to surface the explanation.

Why can't we (or AI) invent ways to explain information that makes it much more digestible? And the solutions simpler?

Why is it necessary to continue to increase complexity when we get better intelligence? Can't we find more simple solutions? Or at least more explainable.

So I guess sci-fi movies were right all along. Nobody in Star Wars knows how hyperspace travel works, it just works. The little robots know everything but almost no human bothers to care. People just carry on with their bickering lives while the bots whiz in the background, and these robots are astonished at human inefficiency every single time, but rarely do anything about it. And people are still people.

“Few-shun drive. Mmmm. Yes.”

https://youtu.be/pfNS2kWf5cY?si=SH6_QC0bCspV-ngz

There are comments that truly reveal a future horrifying and true. Few of them. But I count yours among them.

But I’d argue also that airplanes already achieve this complexity to some degree as well as microprocessors.