John David Pressman makes the case that "predicting the next token" is at the heart of intelligence, since it is a fully general task that requires a system to infer and model the causal structure of the world. This has an intriguing implication: if an AI is trained on its own output, it is incentivized to model itself, or in other words: to become self-aware.
