That won't work. It's easy to get the model to say "I don't know" with the correct prompt, but since the model doesn't even have "knowing" in it, it's just outputting "I don't know" based on a random roll of the probability of its training text having someone said "I don't know". The text "I don't know" won't actually correspond to whether the model knows something or not.

And while we can get into a lengthy and philosophical debate about what it takes to "know" something, my previous paragraph is fairly robust to any sensible definition of "knowing". Write your favorite definition of "knowing" something, then look at the architecture of what GPT actually is on the inside, and tell me if it can actually "know" something based on that architecture. You can of course write the more-or-less begging the question "knowing is a matter of producing correct text when prompted about some fact", but I would have numerous questions around applying that definition of "knowing" to anything other than GPT, or what it means when GPT confidently confabulates something. Don't forget to write your definition and do your analysis in the context not just of GPT outputting the correct capital of Oregon when prompted, but the way it will confidently discuss all sorts of things that don't exist. Your definition should be able to account for some sort of difference between confidently outputting correct data and the way it will equally confidently output complete fiction, and indicate some manner in which GPT has some sort of state difference that indicates it is somehow "aware" of when it is doing one or the other. Because I would say if it can't "tell" if it's confidently emitting facts or confidently emitting fiction that there is a very important and real sense it doesn't really "know" the facts, either. (And I absolutely would apply that standard to humans without question; if you can't tell if you're making stuff up or not, you don't know whatever it is you're talking about.)