I agree with the author, and would like to offer another framing to aid understanding:
1. We've invented a make-documents-bigger algorithm that is trained on lots of documents, to figure out what "fits next".
2. A bunch of systems have been built where there is one document that looks like a movie-script where BotCharacter and UserCharacter are talking, and the LLM is tasked with "make it a bit longer."
3. So you are not talking to an AI. You are simply being fed lines from a character in a work of fiction. The things you type are being inserted as the dialogue for UserCharacter each time.
With that framing, it's easier to see things like:
* "Hallucinations" are not qualitatively different from normal use. It's a digital-daydream the whole time, and people only complain when they don't like where the dream-story is going.
* "Prompt injection" is not qualitatively different from normal use either. All text is ultimately part of the same generic document, regardless of how it got there. The people running the bot just don't like it when it wanders from their original prompt/intent.
* The character is not the author. We can have a story where a character named "ChatGPT" is a cheerful sponge living in a pineapple under the sea that enjoys the smell of grilled meat, but that doesn't mean the actual algorithm is any of those things.
