Everything in their input is tokenized. Asking it to count is like asking a person born blind to paint and complaining they didn't get the colors quite right.
You could train an AI on ASCII or Unicode, but it would likely take 100 times the compute resources for similar performance on everything else. Tokenized input is really efficient.
I wonder if that could be useful, to make AI-resistant CAPTCHA's...
In human terms - think of ChatGPT as a very well-born person, sadly with profound brain issues, who you are meeting at a fancy cocktail hour. His family has spent years and $millions getting him to the point of looking & sounding pretty good, within that very limited social context. But he's never actually made a PB&J sandwich. Let alone gotten dressed and gone to the grocery store by himself.
Over cocktails, Mr. ChatGPT seems darned impressive. But that's because your "is this guy mentally fit?" instincts - developed and tuned in interactions with normal people - naturally jump to flawed conclusions.
2. It's probably not very well represented in the training data. Humans don't often ask other humans to count to a million. Asking why an AI can't count to a million seems similar to asking a human why they can't easily recite the alphabet backwards (unless they've tried it before, or they really think it through ie count forwards to get each prior letter, or they use a 'tool' e.g. writing it down).
This is where people get frustrated with AI, because a human can and would break down the problem, knowing what they know, so they could get the right answer. When the public, who doesn't understand how AI works, is told ChatGPT can do all this amazing stuff and is passing these advanced tests, they get very frustrated when it can't simply count in the way a decades old calculator can.
