Very true! I'm surprised there isn't some well-known trick of scientific process to screen this out, actually.
One that comes to my mind pretty quickly: after the first test to collect people's "novel" results (the one that'll be used as input for the final score), have them take 2-5 more tests with the same tasks (enough so that the questions are utterly non-novel by the end.) Show them the correct answers for the problems on the test they just did between each session.
Now take the scores from all the tests (including the first one), and fit a curve to them. The second derivative at the point of the first test then (might!) represent the novelty they were experiencing at that point. Now figure out how much extra novelty they experienced compared to an average member of the population, and use that as a coefficient to the final IQ score from the first test.