Yes, I think it's unfortunate that you decided to muddle the discussion by introducing the discussion of LLMs in your previous comment, and by conflating them with AGI, in the sense of strongly superhuman AI, which is OpenAI's objective. I worked pretty hard to unmuddle them in my comment.

The questions you are bringing up about the possible limits of the LLM approach are interesting open research questions, and while I really doubt your implicit claim to have resolved them, they are ultimately irrelevant to the topic at hand, which, I will remind you, is the astounding novelty of the situation where

> many companies operating in the public eye are basically stating "We are creating a digital god, an instrument more powerful than any nuclear weapon" and raising billions to do it, and nobody bats an eye...

Note that there is nothing about LLMs in this proposition, and the particular company we're implicitly most focused on—OpenAI—has already developed a number of well-known models that aren't LLMs and plans to keep doing so.

None of my argument depends on the AI being sentient.

You are surely correct that there are weaker imaginable AIs than the strongly superhuman AI that OpenAI and I are talking about which would still be more powerful than nuclear weapons, but they are more debatable. For example, whether discovering new physics would permit the construction of new, more powerful weapons is debatable; it didn't help Archimedes or Tipu Sultan. So discussing such weak claims is likely to end up off in the weeds of logistics and speculation about exactly what kind of undiscovered physics and math would come to light. Instead, I focused on the most obviously correct ways that strongly superhuman AI would be more powerful than nuclear weapons.

These may not be the most practically important ways. Maybe any strongly superhuman AI would immediately discover a way to explode the sun, or to control people's minds, or to build diamondoid molecular nanotechnology, or to genetically engineer super-plagues, or to collapse the false vacuum. Any of those would make nuclear weapons seem insignificant. But claims like those are much more uncertain than the very simple question before us: whether what OpenAI is trying to develop would be more powerful than nuclear weapons. Obviously it would be, by my reasoning in the grandparent comment, even if this isn't a false vacuum, if the sticky fingers problem makes diamondoid nanotechnology impossible, if people's minds are inherently uncontrollable, etc. So we don't need to resolve those other, more difficult questions in order to do the much easier task of ranking OpenAI's objective relative to nuclear weapons.