There are a ton of extremely Hard problems to solve there that we are not likely going to solve.

One: English is terribly non-prescriptive. Explaining an algorithm is incredibly laborious in spoken language and can contain many ambiguous errors. Try reading Euclid’s Elements. Or really any pre-algebra text and reproduce its results.

Fortunately there’s a solution to that. Formal languages.

Now LLMs can somewhat bridge that gap due to how frequently we write about code. But it’s a non-deterministic process and hallucinations are by design. There’s no escaping the fact that an LLM is making up the code it generates. There’s nothing inside the machine that is understanding what any of the data it’s manipulating means or how it affects the system it’s generating code for.

And it’s not even a tool.

Worse, we can’t actually ship the code that gets generated without a human appendage to the machine to take the fall for it if there are any mistakes in it.

If you’re trying to vibe code an operating system and have no idea what good OS design is or what good code for such a system looks like… you’re going to be a bad appendage for the clanker. If it could ship code on its own the corporate powers that be absolutely would fire all the vibe coders and you’d never work again.

Vibe coding is turning people into indentured corporate servants. The last mile delivery driver of code. Every input surveilled and scrutinized. Output is your responsibility and something you have little control over. You learn nothing when the LLM gives you the answer because you’ll forget it tomorrow. There’s no joy in it either because there is no challenge and no difficulty.

I think what pron is leading to is that there’s no need to imagine what these machines could potentially do. I think we should be looking at what they actually do, who they’re doing it to, and who benefits from it.

Yes, but perhaps technology can't plateau beyond vibe coding but below "the machine does everything", not because technology doesn't plateau but because that point doesn't exist. Technology could plateau before both or after both.

All the current indicators are that AI will plateau far beyond any human capability.

Do you have evidence or empirical arguments to the contrary?

It’s not, it’s horrifying.

But there doesn’t seem to be any off ramp, given the incentives of our current economic system.

That's actually a terrific example, because the company already had a working, field-tested manually controlled version of their product, but they wanted a computer-controlled version to relieve radiologists of all of the drudgery and opportunities for error in the manual setup. So the design spec was to "faithfully replicate the manual setup procedures in a microcontroller," which was achieved. The problem was that the UI was utter dogshit, the training materials were poor (you can't have Marketing say "hey this computer handles all the details for you!" and then expect customers to pore page-by-page through the reference manual), and most clinics weren't really enforcing or QAing the operator training anyway. And since "the computer has a handle on it," the radiologists and senior techs who were supposed to be second-checking everything basically abdicated that responsibility, driven in part I'm sure by management that expected them to now have more "free hours" for other activities that Make Line Go Up. And nobody really raised a flag that this seemed troubling until multiple children died, and at the end of the day a lot of people could say "well, it wasn't my fault, I did what I was told to do" and sort of believe it.

If anyone doesn't think all of that is going to happen again on a massive scale as companies embrace GenAI, HAHAHAHAHAHAHA. AI will change a lot of things, but it won't fix human nature.