> Haven't the latest LLMs shown us that a neural net trained to "just string together readable words" lead to at least simple intelligence?
> It feels like we just keeping shifting the goal posts.
Defining "intelligence" is and always has been a bit of a trap. Many non-sapient things exhibit intelligent behavior - crows use tools, Wikipedia has boundless knowledge, and paper contains traces of written intelligence. It's not hard to reconcile ChatGPT with our world, it's merely hard to use it as an analog for humanity. Language is indeed linked with anthropology, but not equivalent to it.
> if I tell a bot "I'm going to turn you off now" and it tries to stop me, that implies it feels fear.
That implies that it has finished a sentence with whatever seemed to come next. The most-likely response to someone using frightening language is to emulate the human responses it's trained on. It thinks a frightened response would satisfy you, and apparently it was right.
