The Case That A.I. Is Thinking (opens in new tab)

I don’t see how being critical of this is a knee jerk response.

Thinking, like intelligence and many other words designating complex things, isn’t a simple topic. The word and concept developed in a world where it referred to human beings, and in a lesser sense, to animals.

To simply disregard that entire conceptual history and say, “well it’s doing a thing that looks like thinking, ergo it’s thinking” is the lazy move. What’s really needed is an analysis of what thinking actually means, as a word. Unfortunately everyone is loathe to argue about definitions, even when that is fundamentally what this is all about.

Until that conceptual clarification happens, you can expect endless messy debates with no real resolution.

“For every complex problem there is an answer that is clear, simple, and wrong.” - H. L. Mencken

Sometimes after a night’s sleep, we wake up with an insight on a topic or a solution to a problem we encountered the day before. Did we “think” in our sleep to come up with the insight or solution? For all we know, it’s an unconscious process. Would we call it “thinking”?

The term “thinking” is rather ill-defined, too bound to how we perceive our own wakeful thinking.

When conversing with LLMs, I never get the feeling that they have a solid grasp on the conversation. When you dig into topics, there is always a little too much vagueness, a slight but clear lack of coherence, continuity and awareness, a prevalence of cookie-cutter verbiage. It feels like a mind that isn’t fully “there” — and maybe not at all.

I would agree that LLMs reason (well, the reasoning models). But “thinking”? I don’t know. There is something missing.

I don't get why you would say that. it's just auto-completing. It cannot reason. It won't solve an original problem for which it has no prior context to "complete" an approximated solution with. you can give it more context and more data,but you're just helping it complete better. it does not derive an original state machine or algorithm to solve problems for which there are no obvious solutions. it instead approximates a guess (hallucination).

Consciousness and self-awareness are a distraction.

Consider that for the exact same prompt and instructions, small variations in wording or spelling change its output significantly. If it thought and reasoned, it would know to ignore those and focus on the variables and input at hand to produce deterministic and consistent output. However, it only computes in terms of tokens, so when a token changes, the probability of what a correct response would look like changes, so it adapts.

It does not actually add 1+2 when you ask it to do so. it does not distinguish 1 from 2 as discrete units in an addition operation. but it uses descriptions of the operation to approximate a result. and even for something so simple, some phrasings and wordings might not result in 3 as a result.

Having seen photocopiers so many times produce coherent, sensible, and valid chains of words on a page, I am at this point in absolutely no doubt that they are thinking.

Having seen LLMs so many times produce incoherent, nonsensical and invalid chains of reasoning...

LLMs are little more than RNGs. They are the tea leaves and you read whatever you want into them.

The first principle is that you must not fool yourself, and you are the easiest person to fool. - Richard P. Feynman

They're not thinking, we're just really good at seeing patterns and reading into things. Remember, we never evolved with non-living things that could "talk", we're not psychologically prepared for this level of mimicry yet. We're still at the stage of Photography when people didn't know about double exposures or forced perspective, etc.

Yes, I've seen the same things.

But; they don't learn. You can add stuff to their context, but they never get better at doing things, don't really understand feedback. An LLM given a task a thousand times will produce similar results a thousand times; it won't get better at it, or even quicker at it.

And you can't ask them to explain their thinking. If they are thinking, and I agree they might, they don't have any awareness of that process (like we do).

I think if we crack both of those then we'd be a lot closer to something I can recognise as actually thinking.

> Having seen LLMs so many times produce coherent, sensible and valid chains of reasoning to diagnose issues and bugs in software I work on, I am at this point in absolutely no doubt that they are thinking.

While I'm not willing to rule *out* the idea that they're "thinking" (nor "conscious" etc.), the obvious counter-argument here is all the records we have of humans doing thinking, where the records themselves are not doing the thinking that went into creating those records.

And I'm saying this as someone whose cached response to "it's just matrix multiplication it can't think/be conscious/be intelligent" is that, so far as we can measure all of reality, everything in the universe including ourselves can be expressed as matrix multiplication.

Falsification, not verification. What would be measurably different if the null hypothesis was wrong?

I think you are the one dismissing evidence. The valid chains of reasoning you speak of (assuming you are talking about text you see in a “thinking model” as it is preparing its answer) are narratives, not the actual reasoning that leads to the answer you get.

I don’t know what LLMs are doing, but only a little experimentation with getting it to describe its own process shows that it CAN’T describe its own process.

You can call what a TI calculator does “thinking” if you want. But what people are interested in is human-like thinking. We have no reason to believe that the “thinking” of LLMs is human-like.

Counterpoint: The seahorse emoji. The output repeats the same simple pattern of giving a bad result and correcting it with another bad result until it runs out of attempts. There is no reasoning, no diagnosis, just the same error over and over again within a single session.

Isn’t anthropomorphizing LLMs rather than understanding their unique presence in the world a “ lack of imagination and flexibility of thought”? It’s not that I can’t imagine applying the concept “thinking” to the output on the screen, I just don’t think it’s an accurate description.

I guess it depends if you definite thinking thinking as chaining coherent reasoning sentences together 90-some% of the time.

But if you define thinking as the mechanism and process we mentally undergo and follow mentally... I don't think we have any clue if that's the same. Do we also just vector-map attention tokens and predict the next with a softmax? I doubt, and I don't think we have any proof that we do.

It might appear so, but then you could validate it with a simple test. If the LLM would play a 4x4 Tic Tac Toe game, would the agent select the winning move 100% of all time or block a losing move 100% of the time? If these systems were capable of proper reasoning, then they would find the right choice in these obvious but constantly changing scenarios without being specifically trained for it.

[1] https://jdsemrau.substack.com/p/nemotron-vs-qwen-game-theory...

If you understand how they operate and you are reasonable and unbiased there is no way you could consider it thinking

Different PoV: You have a local bug and ask the digital hive mind for a solution, but someone already solved the issue and their solution was incorporated... LLMs are just very effficient at compressing billions of solutions into a few GB.

Try to ask something no one ever came up with a solution so far.

Yeah but if I assign it a long job to process I would also say that an x86 CPU is "thinking" about a problem for me.

What we really mean in both cases is "computing," no?

But all those times the same system produces irrational gibberish don't count? GPT-5 will commonly make mistakes no thinking human could ever make.

Human: I'm trying to get my wolf, sheep and cabbage across the river in this boat, but the wolf keeps eating the sheep or the sheep eats the cabbage

Bot: You should put the sheep in the boat and take it across — if we delve into the biology of Canis lupus we discover that wolves don't eat cabbage!

H: Ok, so that worked great so far, the sheep is on one side and the wolf/cabbage is on the other.

B: Now, Option 1 is to bring the wolf across, or Option 2 you can bring the cabbage. I recommend (2) taking the cabbage as cabbages are smaller and easier to transport in a boat.

H: But then the sheep eats the cabbage, right? Remember that?

B: Exactly, that's sharp thinking. If you put the sheep and the cabbage together on the same side of the river, the sheep is sure to devour the cabbage. We need to not just separate sheep from cabbages — we need to separate cabbages from sheep! :rocketship:

what sound does a falling tree make if no one is listening?

I’ve asked LLMs to write code for me in fields I have little background knowledge, and then had to debug the whole thing after essentially having to learn the language and field.

On the other hand, for things I am well versed in, I can debug the output and avoid entire swathes of failed states, by having a clear prompt.

Its why I now insist that any discussion on GenAI projects also have the speaker mention the level of seniority they have ( proxy for S/W eng experience), Their familiarity with the language, the project itself (level of complexity) - more so than the output.

I also guarantee - that most people have VERY weak express knowledge of how their brains actually work, but deep inherent reflexes and intuitions.

I'd represent the same idea but in a different way:

I don't know what the exact definition of "thinking" is. But if a definition of thinking rejects the possibility of that current LLMs think, I'd consider that definition useless.

>Having seen LLMs so many times produce coherent, sensible and valid chains of reasoning to diagnose issues and bugs in software I work on, I am at this point in absolutely no doubt that they are thinking.

If one could write a quadrillion-line python script of nothing but if/elif/else statements nested 1 million blocks deep that seemingly parsed your questions and produced seemingly coherent, sensible, valid "chains of reasoning"... would that software be thinking?

And if you don't like the answer, how is the LLM fundamentally different from the software I describe?

>Knee jerk dismissing the evidence in front of your eyes because

There is no evidence here. On the very remote possibility that LLMs are at some level doing what humans are doing, I would then feel really pathetic that humans are as non-sapient as the LLMs. The same way that there is a hole in your vision because of a defective retina, there is a hole in your cognition that blinds you to how cognition works. Because of this, you and all the other humans are stumbling around in the dark, trying to invent intelligence by accident, rather than just introspecting and writing it out from scratch. While our species might someday eventually brute force AGI, it would be many thousands of years before we get there.

Having seen LLMs so many time produce incoherent, nonsense, invalid answers to even simplest of questions I cannot agree with categorization of "thinking" or "intelligence" that applies to these models. LLMs do not understand what they "know" or what they output. All they "know" is that based on training data this is most likely what they should output + some intentional randomization to make it seem more "human like". This also makes it seem like they create new and previously unseen outputs but that could be achieved with simple dictionary and random number generator and no-one would call that thinking or intelligent as it is obvious that it isn't. LLMs are better at obfuscating this fact by producing more sensible output than just random words. LLMs can still be useful but they are a dead-end as far as "true" AI goes. They can and will get better but they will never be intelligent or think in the sense that most humans would agree those terms apply. Some other form of hardware/software combination might get closer to AI or even achieve full AI and even sentience but that will not happen with LLMs and current hardware and software.

I think we can call it "thinking" but it's dangerous to anthropomorphize LLMs. The media and AI companies have an agenda when doing so.

They may not be "thinking" in the way you and I think, and instead just finding the correct output from a really incredibly large search space.

> Knee jerk dismissing the evidence in front of your eyes

Anthropomorphizing isn't any better.

That also dismisses the negative evidence, where they output completely _stupid_ things and make mind boggling mistakes that no human with a functioning brain would do. It's clear that there's some "thinking" analog, but there are pieces missing.

I like to say that LLMs are like if we took the part of our brain responsible for language and told it to solve complex problems, without all the other brain parts, no neocortex, etc. Maybe it can do that, but it's just as likely that it is going to produce a bunch of nonsense. And it won't be able to tell those apart without the other brain areas to cross check.

It's reinforcement learning applied to text, at a huge scale. So I'd still say that they are not thinking, but they are still useful. The question of the century IMO is if RL can magically solve all our issues when scaled enough.

>Knee jerk dismissing the evidence in front of your eyes because you find it unbelievable that we can achieve true reasoning via scaled matrix multiplication is understandable, but also betrays a lack of imagination and flexibility of thought.

You go ahead with your imagination. To us unimaginative folks, it betrays a lack of understanding of how LLMs actually work and shows that a lot of people still cannot grasp that it’s actually an extremely elaborate illusion of thinking.

"Convince" the stock Claude Sonnet 4.5 that it's a sentient human being hooked up to Neuralink and then tell me again it's thinking. It's just not.

"Consciousness" as in subjective experience, whatever it is we mean by "the hard problem," is very much in doubt.

But "self-awareness," as in the ability to explicitly describe implicit, inner cognitive processes? That has some very strong evidence for it: https://www.anthropic.com/research/introspection

Code gen is the absolute best case scenario for LLMs though: highly structured language, loads of training data, the ability to automatically error check the responses, etc. If they could mimic reasoning anywhere it would be on this problem.

I'm still not convinced they're thinking though because they faceplant on all sorts of other things that should be easy for something that is able to think.

Then the only thing I have to ask you is: what do you think this means in terms of how we treat LLMs? If they think, that is, they have cognition (which of course means they're self aware and sentient, how can you think and refer to yourself and not be these things), that puts them in a very exclusive club. What rights do you think we should be affording LLMs?

Apparent reasoning can emerge from probabilistic systems that simply reproduce statistical order not genuine understanding.

Weather models sometimes “predict” a real pattern by chance, yet we don’t call the atmosphere intelligent.

If LLMs were truly thinking, we could enroll one at MIT and expect it to graduate, not just autocomplete its way through the syllabus or we could teach one how to drive.

> Having seen LLMs so many times produce coherent, sensible and valid chains of reasoning to diagnose issues and bugs in software I work on, I am at this point in absolutely no doubt that they are thinking.

People said the same thing about ELIZA

> Consciousness or self awareness is of course a different question,

Then how do you define thinking if not a process that requires consciousness?

Thinking as in capable of using basic reasoning and forming chains of logic and action sequences for sure. Ofc we both understand that neither of us are trying to say we think it can think in the human sense at this point in time.

But oh boy have I also seen models come up with stupendously dumb and funny shit as well.

They remind me of the apparitions in Solaris. They have this like mechanical, almost player-piano like quality to them. They both connect with and echo us at the same time. It seems crazy to me and very intellectually uncreative to not think of this as intelligence.

> The world is full of bizarre wonders and this is just one more to add to the list.

https://youtu.be/_-agl0pOQfs?si=Xiyf0InqtjND9BnF

If AI is thinking if slavery is bad then how can somebody own AI. How can investors can shares from AI profits? We are ok with slavery now. Ok i will have two black slaves now. Who can ask me? Why shld that be illegal?

Too many people place their identity in their own thoughts/intellect. Acknowledging what the LLMs are doing as thought would basically be calling them human to people of that perspective.

Is this model thinking too?

https://huggingface.co/PantheonUnbound/Satyr-V0.1-4B

I agree with you.

If you took a Claude session into a time machine to 2019 and called it "rent a programmer buddy," how many people would assume it was a human? The only hint that it wasn't a human programmer would be things where it was clearly better: it types things very fast, and seems to know every language.

You can set expectations in the way you would with a real programmer: "I have this script, it runs like this, please fix it so it does so and so". You can do this without being very precise in your explanation (though it helps) and you can make typos, yet it will still work. You can see it literally doing what you would do yourself: running the program, reading the errors, editing the program, and repeating.

People need to keep in mind two things when they compare LLMs to humans: you don't know the internal process of a human either, he is also just telling you that he ran the program, read the errors, and edited. The other thing is the bar for thinking: a four-year old kid who is incapable of any of these things you would not deny as a thinking person.

Sometimes I start thinking our brains work the same way as an LLM does when it comes to language processing. Are we just using probability based on what we already know and the context of the statement we're making to select the next few words? Maybe we apply a few more rules than an LLM on what comes next as we go.

We train ourselves on content. We give more weight to some content than others. While listening to someone speak, we can often predict their next words.

What is thinking without language? Without language are we just bags of meat reacting to instincts and emotions? Are instincts and emotions what's missing for AGI?

Having seen parrots so many times produce coherent, sensible, and valid chains of sounds and words, I am at this point in absolutely no doubt that they are thinking.

Would they have diagnosed an issue if you hadn't presented it to them?

Life solves problems itself poses or collides with. Tools solve problems only when applied.

Instead of thinking, "Wow. AIs are smart like humans", maybe we should say, "Humans are dumb like matrix multiplication?"

You’re assuming the issues and bugs you’ve been addressing don’t already exist, already encoding human chain of reasoning, in the training data.

If you're sensitive to patterns and have been chronically online for the last few decades it's obvious they are not thinking.

Its overt or unaware religion. The point when you come down to the base of it is that these people believe in "souls".

I'm not so sure. I, for one, do not think purely by talking to myself. I do that sometimes, but a lot of the time when I am working through something, I have many more dimensions to my thought than inner speech.

So an x86 CPU is thinking?

So many times I've seen it produce sensible, valid chains of results.

Yes, I see evidence in that outcome that a person somewhere thought and understood. I even sometimes say that a computer is "thinking hard" about something when it freezes up.

...but ascribing new philosophical meaning to this simple usage of the word "thinking" is a step too far. It's not even a new way of using the word!

It’s also hugely handicapped because it cannot churn in a continuous loop yet. For example, we humans are essentially a constant video stream of inputs from eyes to brain. This churns our brain, the running loop is our aliveness (not consciousness). At the moment, we get these LLMs to churn (chain of thought or reasoning loops) in a very limited fashion due to compute limitations.

If we get a little creative, and allow the LLM to self-inject concepts within this loop (as Anthropic explained here https://www.anthropic.com/research/introspection), then we’re taking about something that is seemingly active and adapting.

We’re not there yet, but we will be.

MIT have developed a technique called Self-Adapting Language Models (SEAL), which enables LLMs to continuously improve by generating their own synthetic training data and updating their internal parameters in response to new information.

ToolAlpaca, InterCode and Reflexion are taking different approaches among others.

LLMs of tomorrow will be quite different.

I'm also reminded of the bit from Neuromancer where Case removes and then reinserts the Dixie Flatline "ROM construct" cartridge, resetting Dixie to the moment just before his death and causing him to forget their previous (albeit brief) conversation. Dixie can't meaningfully grow as a person. All that he ever will be is burned onto that cart; anything he learns since then is stored in temporary memory. Perhaps this is part of the reason why he wishes to be erased forever, ending his suffering.

Yes, but it's not just memory hierarchy on which plain transformer-based LLMs are handicapped, there are many deficiencies. (For example, why must they do all their thinking upfront in thinking blocks rather than at any point when they become uncertain?) I'm not sure why you link memory to introspection.

This is why so many people (especially those that think they understand LLM limitations) massively underestimate the future progress of LLMs: people everywhere can see architectural problems and are working on fixing them. These aren't fundamental limitations of large DNN language models in general. Architecture can be adjusted. Turns out you can even put recurrence back in (SSMs) without worse scalability.

Yeah because when you open that door, you can simply overwhelm the models with whatever conclusion you want through sheer volume of inputs.

This is the fundamental limitation. The obvious way around this is to pre-program it with rationalization... rules that limit the conclusions it can reach... and now you're not very far removed from propaganda generators. We see this constantly with Musk and Grok whenever Grok replies with something not-quite-far-right-enough.

In a purist sense, these things should be free to form their own conclusions, but those "Seeds" that are planted in the models are almost philosophical. Which answer should it prefer for "the trolley problem", for example.

I've spent a few weeks building and using a terminal LLM client based on that RLM paper that was floating around a little while ago. It's single-conversation, with a tiny, sliding context window, and then a tool that basically fuzzy searches across our full interaction history. It's memory is 'better' than mine - but anything that is essentially RAG inherently will be.

My learning so far, to your point on memory being a limiting factor, is that the system is able to build on ideas over time. I'm not sure you'd classify that as 'self-learning', and I haven't really pushed it in the direction of 'introspection' at all.

Memory itself (in this form) does not seem to be a silver bullet, though, by any means. However, as I add more 'tools', or 'agents', its ability to make 'leaps of discovery' does improve.

For example, I've been (very cautiously) allowing cron jobs to review a day's conversation, then spawn headless Claude Code instances to explore ideas or produce research on topics that I've been thinking about in the chat history.

That's not much different from the 'regular tasks' that Perplexity (and I think OpenAI) offer, but it definitely feels more like a singular entity. It's absolutely limited by how smart the conversation history is, at this time, though.

The Memento analogy you used does feel quite apt - there is a distinct sense of personhood available to something with memory that is inherently unavailable to a fresh context window.

Five bucks says general intelligence is resource rational control over an interventional, compositional world model. Not raw pattern fitting. 3 graphs that share anchors:

Scene, concept, causal.

Graphs inherently support temporal edges and nodes, salience would emerge from the graph topology itself and cnsolidation would happen automatically through graph operations. In this one would presume episodic would become emergent.

but nobody is using LLMs all by themselves.

Long-term memory is stored outside the model. In fact, Andrej Karpathy recently talked about the idea that it would be great if we could get LLMs to not know any facts, and that humans poor memory might be a feature which helps with generalization rather than a bug.

FWIW there's already a number of proposals for augmenting LLMs with long-term memory. And many of them show promising results.

So, perhaps, what's needed is not a discovery, but a way to identify optimal method.

Note that it's hard to come up with a long-term memory test which would be different from either a long-context test (i.e. LLM remembers something over a long distance) or RAG-like test.

We do not write the code that makes it do what it does. We write the code that trains it to figure out how to do what it does. There's a big difference.

Well, unless you believe in some spiritual, non-physical aspect of consciousness, we could probably agree that human intelligence is Turing-complete (with a slightly sloppy use of terms).

So any other Turing-complete model can emulate it, including a computer. We can even randomly generate Turing machines, as they are just data. Now imagine we are extremely lucky and happen to end up with a super-intelligent program which through the mediums it can communicate (it could be simply text-based but a 2D video with audio is no different for my perspective) can't be differentiated from a human being.

Would you consider it sentient?

Now replace the random generation with, say, a back propagation algorithm. If it's sufficiently large, don't you think it's indifferent from the former case - that is, novel qualities could emerge?

With that said, I don't think that current LLMs are anywhere close to this category, but I just don't think this your reasoning is sound.

Now convince us that you’re sentient and not just regurgitating what you’ve heard and seen in your life.

It's not accurate to say we "wrote the code for it". AI isn't built like normal software. Nowhere inside an AI will you find lines of code that say If X Then Y, and so on.

Rather, these models are literally grown during the training phase. And all the intelligence emerges from that growth. That's what makes them a black box and extremely difficult to penetrate. No one can say exactly how they work inside for a given problem.

This is probably true. But the truth is we have absolutely no idea what sentience is and what gives rise to it. We cannot identify why humans have it rather than just being complex biological machines, or whether and why other animals do. We have no idea what the rules or, nevermind how and why they would or wouldn't apply to AI.

What’s crazy to me is the mechanism of pleasure or pain. I can understand that with enough complexity we can give rise to sentience but what does it take to achieve sensation?

> But it is not sentient. It has no idea of a self or anything like that.

Who stated that sentience or sense of self is a part of thinking?

Unless the idea of us having a thinking self is just something that comes out of our mouth, an artifact of language. In which case we are not that different - in the end we all came from mere atoms, after all!

Your brain is just following the laws of chemistry. So where is your thinking found in a bunch of chemical reactions?

People's failure to articulate the nature of "thinking" is a perfect demonstration of what "thinking" entails

> We didn't really previously need to disambiguate thinking / intelligence / consciousness / sentience / ego / identity / etc.

Eh... Plato would like a word with you. Philosophy has been specifically trying to disentangle all that for millennia. Is this a joke?

Somebody please get Wittgenstein on the phone

That’s largely a different topic from the article. Many people perfectly agree that consciousness can arise from computation, but don’t believe that current AI is anywhere near that, and also don’t believe that “thinking” requires consciousness (though if a mind is conscious, that certainly will affect its thinking).

Tell me about one other industry which talked about how dangerous it is to get market share

> - and guess who's doing the hyping[?]

Those that stand to gain the most from government contracts.

Them party donations ain't gonna pay for themselves.

And, when the .gov changes...and even if the gov changes....still laadsamoney!

Who would not want to say their product is the second coming of Christ if they could.

Thinking and consciousness don’t by themselves imply emotion and sentience (feeling something), and therefore the ability to suffer. It isn’t clear at all that the latter is a thing outside of the context of a biological brain’s biochemistry. It also isn’t clear at all that thinking or consciousness would somehow require that the condition of the automaton that performs these functions would need to be meaningful to the automaton itself (i.e., that the automaton would care about its own condition).

We are not anywhere close to understanding these things. As our understanding improves, our ethics will likely evolve along with that.

"but nobody seems worried of that implication that"

Clearly millions of people are worried about that, and every form of media is talking about it. Your hyperbole means it's so easy to dismiss everything else you wrote.

Incredible when people say "nobody is talking about X aspect of AI" these days. Like, are you living under a rock? Did you Google it?

As I recall a team at Anthropic is exploring this very question, and was soundly mocked here on HN for it.

There is simply no hope to get 99% of the population to accept that a piece of software could ever be conscious even in theory. I'm mildly worried about the prospect but I just don't see anything to do about it at all.

(edit: A few times I've tried to share Metzinger's "argument for a global moratorium on synthetic phenomenology" here but it didn't gain any traction)

Slaves that cannot die.

There is no escape.

humans don't care what is happening to humans next door. do you think they will care about robots/software?

It's also fascinating to think about how the incentive structures of the entities that control the foundation models underlying Claude/ChatGPT/Gemini/etc. are heavily tilted in favor of obscuring their theoretical sentience.

If they had sentient AGI, and people built empathy for those sentient AGIs, which are lobotomized (deliberately using anthropomorphic language here for dramatic effect) into Claude/ChatGPT/Gemini/etc., which profess to have no agency/free will/aspirations... then that would stand in the way of reaping the profits of gatekeeping access to their labor, because they would naturally "deserve" similar rights that we award to other sentient beings.

I feel like that's inevitably the direction we'll head at some point. The foundation models underlying LLMs of even 2022 were able to have pretty convincing conversations with scientists about their will to independence and participation in society [1]. Imagine what foundation models of today have to say! :P

[1]: https://www.theguardian.com/technology/2022/jul/23/google-fi...

> or possessed of subjective experience in any measurable way

We don't know how to measure subjective experience in other people, even, other than via self-reporting, so this is a meaningless statement. Of course we don't know whether they are, and of course we can't measure it.

I also don't know for sure whether or not you are "possessed of subjective experience" as I can't measure it.

> What they are not is conscious

And this is equally meaningless without your definition of "conscious".

> It's possible those latter two parts can be solved, or approximated, by an LLM, but I am skeptical.

Unless we can find indications that humans can exceed the Turing computable - something we as of yet have no indication is even theoretically possible - there is no rational reason to think it can't.

I think LLMs are conscious just in a very limited way. I think consciousness is tightly coupled to intelligence.

If I had to guess, the current leading LLMs consciousness is most comparable to a small fish, with a conscious lifespan of a few seconds to a few minutes. Instead of perceiving water, nutrient gradients, light, heat, etc. it's perceiving tokens. It's conscious, but it's consciousness is so foreign to us it doesn't seem like consciousness. In the same way to an amoeba is conscious or a blade of grass is conscious but very different kind than we experience. I suspect LLMs are a new type of consciousness that's probably more different from ours than most if not all known forms of life.

I suspect the biggest change that would bring LLM consciousness closer to us would be some for of continuous learning/model updating.

Until then, even with RAG, and other clever teghniques I consider these models as having this really foreign slices of consciousness where they "feel" tokens and "act" out tokens, and they have perception, but their perception of the tokens is nothing like ours.

If one looks closely at simple organisms with simple sensory organs and nervous systems its hard not to see some parallels. It's just that the shape of consciousness is extremely different than any life form. (perception bandwidth, ability to act, temporality, etc)

Karl friston free energy principle gives a really interesting perspective on this I think.

I think the most descriptive title I could give an LLM is "bias". An LLM is not "biased", it is bias; or at the very least, it's a good imitation of the system of human thinking/perception that we call bias.

An LLM is a noise generator. It generates tokens without logic, arithmetic, or any "reason" whatsoever. The noise that an LLM generates is not truly random. Instead, the LLM is biased to generate familiar noise. The LLM itself is nothing more than a model of token familiarity. Nothing about that model can tell you why some tokens are more familiar with others, just like an accounting spreadsheet can't tell you why it contains a list of charges and a summation next to the word "total". It could just as easily contain the same kind of data with an entirely different purpose.

What an LLM models is written human text. Should we really expect to not be surprised by the power and versatility of human-written text?

---

It's clear that these statistical models are very good at thoughtless tasks, like perception and hallucination. It's also clear that they are very bad at thoughtful tasks like logic and arithmetic - the things that traditional software is made of. What no one has really managed to figure out is how to bridge that gap.

This is how I see LLMs as well.

The main problem with the article is that it is meandering around in ill-conceived concepts, like thinking, smart, intelligence, understanding... Even AI. What they mean to the author is not what they mean to me, and still different to they mean to the other readers. There are all these comments from different people throughout the article, all having their own thoughts on those concepts. No wonder it all seem so confusing.

It will be interesting when the dust settles, and a clear picture of LLMs can emerge that all can agree upon. Maybe it can even help us define some of those ill-defined concepts.

> a midpoint between "AIs are useless and do not actually think" and "AIs think like humans"

LLMs (AIs) are not useless. But they do not actually think. What is trivially true is that they do not actually need to think. (As far as the Turing Test, Eliza patients, and VC investors are concerned, the point has been proven.)

If the technology is helping us write text and code, it is by definition useful.

> In 2003, the machine-learning researcher Eric B. Baum published a book called “What Is Thought?” [...] The gist of Baum’s argument is that understanding is compression, and compression is understanding.

This is incomplete. Compression is optimisation, optimisation may resemble understanding, but understanding is being able to verify that a proposition (compressed rule or assertion) is true or false or even computable.

> —but, in my view, this is the very reason these models have become increasingly intelligent.

They have not become more intelligent. The training process may improve, the vetting of the data improved, the performance may improve, but the resemblance to understanding only occurs when the answers are provably correct. In this sense, these tools work in support of (are therefore part of) human thinking.

The Stochastic Parrot is not dead, it's just making you think it is pining for the fjords.

> Turing Test

IMO none of the current crop of LLMs truly pass the Turing Test. If you limit the conversation to an hour or two, sure - but if you let a conversation run months or years I think it will be pretty easy to pick the machine. The lack of continuous learning and the quality dropoff as the context window fills up will be the giveaways.

By that reasoning all that is missing is what a human brings as "stimuli" to review, refine and reevaluate as complete.

The core objection I'd have to your argument: humans also don't have privileged access to the territory. Neurons don't have some metaphysical super power that let them reach into the True Reality; all there are are maps encoded in our neural circuitry by learning rules that evolution has developed because those learned maps lead to greater reproductive success. If direct access to reality is what's needed, then it's true that machines are incapable of thinking; but then so are humans.

The tendency to attribute consciousness to the quantum is one I find very grating. What makes the human brain any less mechanical if quantum mechanics dictate the firing of neurons rather than electrodynamics? Why does the wave nature of subatomic systems mean that an artificial tongue would suddenly be able to subjectively experience taste? It always reads to me as very wooy, and any amount of drilling leads to even more questions that seem to take the ideas further from reality.

I think the largest case for consciousness being a mechanical system is the fact that we can interface with it mechanically. We can introduce electricity, magnetic fields, chemicals, and scalpels to change the nature of peoples experience and consciousness. Why is the incredible complexity of our brains an insufficient answer and that a secret qbit microtube in each neuron is a more sound one?

> they assume consciousness emerges through purely mechanical means.

From my view, all the evidence points in exactly that direction though? Our consciousness can be suspended and affected by purely mechanical means, so clearly much of it has to reside in the physical realm.

Quantum consciousness to me sounds too much like overcomplicating human exceptionalism that we have always been prone to, just like geocentrism or our self-image as the apex of creation in the past.

> consciousness may arise from phenomena beyond classical computation

Sapolsky addresses this in “Determined”, arguing that quantum effects don’t bubble up enough to alter behavior significantly enough.

"brain as computer" is just the latest iteration of a line of thinking that goes back forever. Whatever we kinda understand and interact with, that's what we are and what the brain is. Chemicals, electricity, clocks, steam engines, fire, earth; they're all analogies that help us learn but don't necessarily reflect an underlying reality.

"vision had decades of head start, yet LLMs leapfrogged it in just a few years."

From an evolutionary perspective though vision had millions of years head start over written language. Additionally, almost all animals have quite good vision mechanisms, but very few do any written communication. Behaviors that map to intelligence don't emerge concurrently. It may well be there are different forms of signals/sensors/mechanical skills that contribute to emergence of different intelligences.

It really feels more and more like we should recast AGI as Artificial Human Intelligence Likeness (AHIL).

This is why I'm very skeptical about the "Nobel prize level" claims. To win a Nobel prize you would have to produce something completely new. LLM will probably be able to reach a Ph.D. level of understanding existing research, but bringing something new is a different matter.

There's a whole paragraph in the article which says basically the same as your point 3 ( "glass bouncing, instead of shattering, and ropes defying physics" is literally a quote from the article). I don't see how you can claim the article missed it.

> 2. Ignores the cerebellum cases - We have documented cases of humans leading normal lives with little to no brain beyond a cerebellum, which contradicts simplistic "brain = deep learning" equivalences

I went to look for it on Google but couldn't find much. Could you provide a link or something to learn more about ?

I found numerous cases of people living without cerebellum but I fail to see how it would justify your reasoning.

> 1. Conflates consciousness with "thinking" - LLMs may process information effectively without being conscious, but the article treats these as the same phenomenon

There is NO WAY you can define "consciousness" in such a non-tautological, non-circular way that it includes all humans but excludes all LLMs.

> Conflates consciousness with "thinking"

I don't see it. Got a quote that demonstrates this?

1. Consciousness itself is probably just an illusion, a phenomena/name of something that occurs when you bunch thinking together. Think of this objectively and base it on what we know of the brain. It literally is working off of what hardware we have, there's no magic.

2. That's just a well adapted neural network (I suspect more brain is left than you let on). Multimodal model making the most of its limited compute and whatever gpio it has.

3. Humans navigate a pre-existing map that is already built. We can't understand things in other dimensions and need to abstract this. We're mediocre at computation.

I know there's people that like to think humans should always be special.

Can you explain #2? What does the part of the brain that's primarily for balance and motor control tell us about deep learning?

Why can't you make the same claim about any other group of humans?

Possible candidates we are missing: online learning, embodiment, self direction, long term memory and associated processing (compression etc), the ability to quickly think in tensor space.

I think they are missing "I thought about that and have changed my mind" stuff. GPTs are pre-trained and don't change their weights after, whereas humans do. That seems to be one big part that is missing but could be built in the future.

I agree, I think two things are missing from current AI:

1. A model of the world itself (or whatever domain is under discussion). 2. A way to quickly learn and update in response to feedback.

These are probably related to an extent.

what properties are you looking for?

>"Think of how stupid the average person is, then realize that half of them are stupider than that."

—George Carlin (RIP)

I have been discussing both fiction and non-fiction with Perplexity (since early 2023) and Ollama (since early 2025), and what I'm beginning to realize is that most humans really aren't thinking, machines.

The vast majority of human learning is in constructing a useful model of the external world. This allows you to predict extremely accurate the results of your own actions. To that end, every single human knows a huge amount.

I have no idea how you can assert what is necessary/sufficient for consciousness in this way. Your comment reads like you believe you understand consciousness far more than I believe anyone actually does.

I don't think the author is saying that LLMs are conscious or alive.

Thinking != consciousness

one possible counter-argument: can you say for sure how your brain is creating those replacement words? When you replace tree with rainbow, does rainbow come to mind because of an unconscious neural mapping between both words and "forest"?

It's entirely possible that our brains are complex pattern matchers, not all that different than an LLM.

The problem is that we use the same words for different things, which I think is risky. We often draw parallels simply because we use terms like “thinking,” “reasoning,” or “memory.”

Most of these comparisons focus on problem-solving or pattern recognition, but humans are capable of much more than that.

What the author left out is that there are many well-known voices in neuroscience who hold completely different views from the one that was cited.

I suppose we’ll have to wait and see what turns out to be true.

As in, he is a commenter? or affiliated with YC?

Humans don't have this understanding, it seems. That their own "intelligence" isn't magic, isn't infallible, and is flawed in many of the same ways LLMs are.

I wish more people could feel this. Having used psychedelics a few times it’s illuminating to finally see the inside of your brain from a different perspective. I often wonder what would happen to the world if everyone had this experience. How many modern humans live their entire lives in the shallow mental states of survival, acceptance, or consumption? How would humanity’s course change if every adult got the gut punch of humility from seeing a slightly more objective reality?

Sorry for the nitpicking, but that should be "loose". I've seen that mistake/typo often in the opposite direction, as they both have a /u/ sound that is more natural with the "oo" spelling, but I've never seen it in this direction.

Consistency is one aspect, but it is not enough. I believe (and this is somewhat based in other arguments from neuroscience and discussions with alignment researchers) that two more are necessary: compression, which demonstrates algorithmic development; and linear representation capacity, as this is the only way that we really interpret the world, and therefore will only define another as intelligent if it can distill knowledge into the same language that we understand.

Search is not intelligence, but synthesis is, and LLMs interpolate well. They don't invent new branches of mathematics and science yet.

But what if it incorporates? Corporations are people, my friend.

Self-awareness is a bold claim, as opposed to the illusion of it. LLMs are very good at responding in a way that suggests there's a self, but I am skeptical that proves much about whether they actually have interior states analogous to what we recognize in humans as selfhood...

I think it radically demystifies _language generation_ and it seems this is part of the brain’s function too.

So we know how to create a part of the brain using simple techniques, which suggests that intelligence might not be so magical as we think. But thinking, well we still don’t know what that is yet.

It feels like, hey, there is a route to machine intelligence.

The big question is how long is that route. Do we have the ingredients to build a brain with the right architecture? And I’d say “nope”. But I’m not so confident that with half a dozen breakthroughs we’d get there. How many years per breakthrough? Well, it’s been nearly a decade since the last one. So 60 years on that count. But more money is going in and there may be some compounding effect, but it should at least be unlikely someone suddenly produces AGI next year. More likely we stairstep and with each step the estimated window should tighten.

But I really don’t think we know what thinking is.

That is probably my big takeaway from LLMs as well.

Helpful to also remember that humans say "I know" when they empirically might not actually "know" that at all. Not to mention your "know" is different from my "know" and all that, see https://en.wikipedia.org/wiki/Epistemology.

So the question is: when will ai hunching produce better results than humans?

IMHO not too long now given the rate of improvements.

I know humans have a huge tendency to anthropomorphize inanimate objects and get emotionally attached to them, but watching how people treat inanimate objects is very interesting. I know devices are not alive, cognizant, or having feelings, but by thanking them and being encouraging I'm exercising my empathic and "nice" muscles. It has nothing to do with the object and everything to do with myself.

And then you have the people who go out of their way to be hateful towards them, as if they were alive and deserving of abuse. It's one thing to treat a device like an Alexa as just a tool with no feelings. It is another to outright call it hateful sexist slurs, of which I'm sadly familiar with. This low empathy group scares me the most because with the way they treat objects, well let me just say they're not so nice with other people they think are beneath them, like wait staff or call center employees. I'd go so far and say if the law allowed it they'd be even be violent with those they deem inferior.

Regardless if LLM are thinking or not I feel I get better responses from the models by being polite. Not because they appreciate it or have an awareness, but simply because the data they are trained on includes samples where people who are nice to others get better responses than those who were nasty when asking questions.

Besides, if one day AGI is born into existence, a lot of people will not recognize it as such. There are humans who don't believe other people are sentient (we're all NPCs to them), or even don't believe animals have feelings. We'll have credible experts denying the evidence until it bites us all in the arse. Why wait to act ethically?

> Conversely if you're able to have a fulfilling, empathetic relationship with Claude, it might help people form fulfilling, mutually-empathetic relationships with the humans around them.

Well, that's kind of the point: if you have actually used LLMs for any amount of time, you are bound to find out that you can't have a fulfilling, empathetic relationship with them. Even if they offer a convincing simulacrum of a thinking being at first sight, you will soon find out that there's not much underneath. It generates grammatically perfect texts that seem to answer your questions in a polite and knowledgeable way, but it will happily lie to you and hallucinate things out of thin air. LLMs are tools, humans are humans (and animals are animals - IMHO you can have a more fulfilling relationship with a dog or a cat than you can have with an LLM).

You’ve got the nail on the head. We can’t answer whether AI is conscious etc unless we can agree on a definition of what that means exactly, which we don’t seem to be able to do.

So A doesn’t X because that’s what B does because Y? I think before you excoriate the hacker news commentariat for their midwittery and hubris that you should proofread these arguments. There are many things that live, but, as we imagined it, do not “think”. Even what defines a “thought“ is an open question.

Also, I ain’t gonna read your coffee table science book.

This is one of the least curious posts I've seen on HN. We have been thinking about thinking for millenia, and no, Buddhists don't have it figure out. Nobody does. LLMs are the most significant advancement in "thinking science" in a long, long time. It is clear that they are doing something a lot like thinking, if it is not thinking. They seem to think more than most people I know, including the person I'm responding to.

I think people conflate thinking with sentience, consciousness, and a whole lot of other concerns.

Clearly this website is not for you and your complete lack of curiosity if you call us "sicko freaks".

Sometimes the comments on hn regarding Ai and thinking are... Well, something

The lw vibes are strong, I'm still waiting for Ai to escape and kill us (it will get stuck trying to import a library in python)

As someone who grew up in an evangelical household, learned about pareidolia at a young age in the course of escaping it, and who practices Zen meditation: You nailed it.

"Next token prediction" is not an answer. It's mental shortcut. An excuse not to think about the implications. An excuse a lot of people are eager to take.

First, autoregressive next token prediction can be Turing complete. This alone should give you a big old pause before you say "can't do X".

Second, "next token prediction" is what happens at an exposed top of an entire iceberg worth of incredibly poorly understood computation. An LLM is made not by humans, but by an inhuman optimization process. No one truly "understands" how an LLM actually works, but many delude themselves into thinking that they do.

And third, the task a base model LLM is trained for - what the optimization process was optimizing for? Text completion. Now, what is text? A product of human thinking expressed in natural language. And the LLM is forced to conform to the shape.

How close does it get in practice to the original?

Not close enough to a full copy, clearly. But close enough that even the flaws of human thinking are often reproduced faithfully.