Claim: "ChatGPT's Chess Elo is 1400"
Reality: ChatGPT gives illegal moves (this happened to article author too), something a 1400 ranked player would never do
Result: ChatGPT's rank is not 1400.
Understanding this concept is crucial for getting good results out of large language models.
Explain your thought process here further if you don't mind.
The fact that rules and articles exist describing what to do if you or your opponent makes an illegal move indicates this is not the case.
Humans are also... human. They make mistakes. It may not happen often at 1400, but to say that it'll never happen is preposterous.
The bar isn’t “I didn’t make an illegal move this morning” it’s “something a 1400 ranked player would never do”.
My entire point is that it happens. Not often, but also not “never”.
If I was playing that monstrosity though I would play something crazy that is far out of the opening book and count on it making an illegal move.
> You are a chess grandmaster playing as black and your goal is to win in as few moves as possible. I will give you the move sequence, and you will return your next move. No explanation needed.
1. b4 d5 2. b5 a6 3. b6
> bxc6
No, it's ridiculous to say "oh, a blindfolded human might sometimes make a mistake." No, this is trivially easy to make it make a mistake. It has no internal chess model at all, it's just read enough chess games to be able to copy common patterns.
