Going meta for a moment, this argument begs the question, assuming the conclusion "Therefore LLMs are not intelligent" in the premises "No intelligence can be ascribed to LLMs".

I'm not convinced it's even possible to come up with a principled, non-circular definition of intelligence (that is, not something like "intelligence is that trait displayed by humans when we...") that would include humans, include animals like crows and octopuses, include a hypothetical alien intelligence, but exclude LLMs.

I'm not arguing that LLMs are intelligent. I'm arguing that the debate is inherently unwinnable.

you seem to be begging the question

almost precisely the same assertions could be made about you with precisely the same degree of justification: you aren't an agent and no intellect can be ascribed to you. you are a device unintelligent agents have made and ascribing you intellect is equally as erroneous

an intelligent agent would have recognized that your argument relies on circular reasoning, but because you are a glorified autocomplete incapable of understanding the meanings of the words you are using, you posted a logically incoherent comment

(of course i don't actually believe that about you. but the justification for believing it about gpt-4 is even weaker)

Why couldn't it be an agent for the short time that it's generating an intelligent looking paragraph?

Do you know what gives rise to consciousness? If not how can we be sure it doesn't arise from a giant pile of linear algebra?

Can any device ever be intelligent, according to you?

The joke is how you decide that the machine isn't an agent. If you believe only meat can be an agent and given the fact that machine isn't meat, it follows machine isn't an agent. The story reverses this chauvinism and shows machines finding an idea of thinking meat absurd for arguably better reason that machines are better fit for information processing than meat.

Yann's explanation here is a pretty high level overview of his understanding of different thought modeling, it isn't really related to how we define intelligence at all and isn't a complete picture. The distinction drawn between System 1 & 2 as explained is more of a limitation in conditions given to the algorithm rather than ability of the algorithm itself (i.e. one could change parameters to allow for unlimited processing time)

Yann may touch in how we define intelligence elsewhere, I haven't deeply studied all of his work. Though I can say that OpenAI has taken to using relative economic value as their analog for comparing intelligence to humans. Personally that definition is pretty gross and offensive, I hope most people wouldn't agree that our intelligence can be directly tied to how much value we can produce in a given economic paradigm.