In human cognition theory, human thinking is governed by two systems: the fast and intuitive System 1 and the slower but more deliberative System 2.
This isn't just pedantry: the initial "System 1 = inference" error makes "System 2 = chain-of-thought" especially flawed. CoT in transformer LLMs helps solve O(n) problems but struggles with O(n^2). The observation that a O(n^2) problem can be broken down into n separate O(n) problems is ultimately due to system 1 reasoning: it is obviously true. But it is only obviously true to smart things like humans and pigeons. Transformers do not seem smart enough to grasp it: system 2 thinking must be "glued together" by tautologies or axioms, and we can only recognize tautologies or discover axioms because of system 1. If the problem is more complex than O(n) these tautologies and axioms must be provided to the LLM, either with a careful prompt or exhaustive data.
Kahneman's book has been largely repudiated on the science. That doesn't mean it isn't a useful way to understand the kinds of errors humans make in decision-making. But it does make the book useless for AI researchers: I believe AGI is well over 200 years away, because going all the way back to Alan Turing AI has simply refused to engage with the challenges of cognitive science, preferring fairy tales which confirm intuitions and trivialize human minds.
[1] https://www.cell.com/trends/cognitive-sciences/abstract/S136... and https://www.psychologytoday.com/intl/blog/a-hovercraft-full-...
I think you’re overthinking this. System 1 thinking as the term is being used by AI researchers means making a fast decision based on reasoning processes that are wired into your brain by evolution. For any task that humans have faced for millions of years this works well. It can also work well for experts in a domain who have practiced a task so many times that their brains have adapted to perform it unconsciously.
System 2 thinking is consciously using explicit reasoning techniques to think through a problem, slowly and rigorously, often in ways that feel unnatural due to our cognitive biases but can solve problems that System 1 is unable to.
The analogy to LLMs is straightforward: LLMs learn to solve many kinds of complex problems during training and encode processes for those specific problems. They can then perform these tasks in a single forward pass through their weights. This is System 1 for LLMs and again, works well for any task that they were exposed to repeatedly during training.
However they don’t generalize to tasks that were not well represented in the training data. Training them to use explicit reasoning strategies instead (System 2) is shown to improve performance and let them solve a broader range of problems.
On the other hand, the reason this idea is sticky is because it matches our conscious experience. In some situations, we respond intuitively. In other situations, we choose to work analytically using tools like research, deliberation, note-taking, etc.
I think it's this second sense in which people are using the term with respect to LLMs. And it's not a terrible analogy.
However, comparing "neural networks" to actual neurons is almost never useful.
> he observation that a O(n^2) problem can be broken down into n separate O(n) problems is ultimately due to system 1 reasoning: it is obviously true.
As the parent of a third grader just learning this stuff, I can assure you it isn't immediately obvious to everyone.
What do you mean by `n` in this case?