The ability to imitate someone's tone (long a staple of satirical writing), or to fool a lot of unread people, no doubt represents a true achievement for LLMs. But people easily fool themselves and believe what they want to believe, so LLMs simply play into the limitations of human models and explanations for the world rather than representing a new form of conscious being.

I think we can see an analogy in craft production versus mass production. Today machines can produce (for example) furniture of very high quality, rivaling or exceeding the output of a craft carpenter or joiner. The mass-produced goods cost less to make and have good-enough quality, and can even sometimes fool people into thinking a skilled person made their table. That doesn't mean we should call those machines carpenters or conclude the machines possess the same skills. With LLMs we see mass production at scale come to writing and some other trades (law, customer support, etc.) that we like to think of as requiring actual skill. Maybe other trades such as programming will also fall to LLM mass production. But that happens because the jobs consisted mainly of rote and repetition in the first place, and require little creativity or intelligence to imitate.

That capacity to fool people with apparently meaningful and creative content may have some value in the market, but I think that value amounts to the same value the market assigns to what currently gets churned out by people working in "content production." Not zero, but not the same value we assign to actual creative writing.

People losing their jobs to ChatGPT don't even call themselves writers, at least not seriously. They produce content, most of it regurgitated the same way ChatGPT does it. If I produced content for online click farms I would worry about so-called AI. I don't think Murakami or Cormac McCarthy have to worry, but James Patterson might get replaced by an LLM (if that didn't happen already -- hard to tell).

In my own profession, some programmers already express fear for their future because Github Copilot and ChatGPT can write code just as good as they can. That should tell them to level up their skills, because if whatever they do can just get automated away by a glorified auto-complete they weren't adding enough value in the first place.