This is true. But I think ChatGPT compared to a human player of equivalent strength(leaving the dubious number 1400 out for now) will probably have a lot more opening knowledge than the human.

So it would be interesting to eliminate all opening knowledge and that way be able to qualitately get at which aspects of chess it's actually good at, which is sucks out, and how much of its strength can be attributed to opening knowledge.

I'm still impressed by this btw. I did not expect this to be possible at all really. But being impressed is not an excuse to ignore methodological flaws. :)