Roger Penrose made an argument like we can know Gödel's theorem is true without being able to prove it but AI can't, but I think you can figure both are guessing in a similar pattern recognising kind of way.