For a UTM, you need something to function as an infinite Turing tape. For example a mechanism that can loop arbitrarily. On a machine with finite resources, running an infinite program will exhaust the machine's resources. But ChatGPT has nothing like that.

As to ChatGPT's memory that's not about UTM expressivity, but another reason why it can't be any kind of computing device.

ChatGPT and other language models of its kind don't have memories. It can retrieve context from earlier prompts because each of your prompts, and the language model's responses to your prompts, are prepended to your latest prompt and then the whole thing passed to the model. It doesn't remember what you said earlier, it just reads it again each time.

Finally, a neural net architecture and a trained model are two different things. For example, you could train one Convolutional neural net to recognise cats, and another to recognise dogs. Suppose you could train a Transformer to simulate a UTM. ChatGPT is not trained to simulate a UTM, it's trained to predict text. So it can't simulate a UTM.

But, after all this discussion I have to ask: why do you say that "it’s pretty safe to bet that ChatGPT is Turing Complete"? What does ChatGPT have in common with cellular automata, for instance?