But did they account for the "I don't know so I'll just randomly pick something" vs the "I should know this, so I'll reason about it" groups?

If you're teaching a subject to someone, you will often come across situations where what is lacking is not knowledge, but confidence or willingness to apply the knowledge. I see that often in my son, and I've seen that at work: People say they don't know, or make a crap guess even in situations where I know that they know the actual answer. Often some prodding or "confidence boosting" will make them produce the real answer very quickly, and often subsequent answers appears to be forthcoming a lot quicker.

It'd be very surprising to me if you can't systematically improve peoples response by increasing their confidence in their ability to answer. The more interesting question to me is by how much, and with how little encouragement.

I think it's due to expectations. Every single time I read about psychology research there are these glaring gaps in methodology that completely nullify their experiments. After losing track of how many psych papers have used statistics wrongly, planned their experiment poorly or had seriously questionable methodology I have lost faith in this field. It's too much work going through a paper every time something reaches the news.

I therefore make the very personal and questionable choice to ignore (most of) them and go straight to the neuroscience people - which in general have a way higher understanding and higher scientific standards.

> ... some commenters always assume (without any evidence) that the results are due to some extremely naive methodological mistake.

Since the article doesn't presume to explain its results, the presence or absence of mistakes is moot. Science isn't about descriptions -- that's metrology. The threshold of science is crossed when someone dares to offer a testable, falsifiable explanation. But in psychology, that rarely happens, and psychology has no central defining theories (explanations) such as are found in scientific fields.