For instance, the % of budget spent on defence by the US question only includes (I assume) the DoD discretionary budget and the Overseas Contingency Budget. It leaves out the VA, the State Dept. Homeland Security, the Justice Dept. , the National Intelligence Program, and the NSF, all of which have defence components.
So in that case it's not so much that someone is wrong if they pick the incorrect answer, it's just that the definition of defence spending is nuanced.
Sites other than Wikipedia show different numbers as well. http://www.cbpp.org/cms/?fa=view&id=1258
The correct answer is almost certainly 100%, but this is not even available as an option. Without any time horizon specified, any value less than 100% is not really meaningful.
Because the goal of each question is to get people to select the wrong answer, the buckets are arbitrarily sized to make guessing difficult, the sources are chosen to be those that have a particular bias, and the questions are worded in ways to make the question misleading on first read.
The main reason this causes me to cast mild aspersions on it is because it's such a missed opportunity. I post this as someone sympathetic to the authors' views, but who is more sympathetic to accurate discourse. People don't know what they don't know, and it's in everyone's best interest to become well-calibrated to their uncertainty. Instead the site will just be ignored by people with opposing political views, and used as back-patting fodder for people who are in support of these political views. It would be much better if it asked a wider variety of questions, scored you based on how close you were rather than simply right or wrong, had ways of signalling and scoring your uncertainty for each question, and had reasonable buckets for each question. I understand that this may not serve the ends of the creators as well, however.
Some examples:
What was the average American college student's college-related debt in 2013? (The actual source is for college GRADUATES, not college students)
How old is the known universe, in years? (One answer is 7000 years, the other is 14 billion years)
What percentage of animal species are headed to extinction in the next century? (This is sourced to Democracy Now! and the exact quote of the source is "Scientists say we are now experiencing the sixth extinction, with up to 50 percent of all living species in danger of disappearing by the end of the century" which is a lot more hedging than the question implies. >10% of the questions on the site are related to percentages of animal extinction [Like what I did there?].)
What percent of U.S. prison inmates are functionally illiterate? (The definition of functionally illiterate is very counterintuitive to the average person, so this question doesn't mean what you think it means)
What percentage of inmates in U.S. federal prisons have mental health problems? (This refers to people who have reported symptoms of mental health problems, not those who were diagnosed as having mental health problems)
These are just the first random questions I clicked on, and by no means exhaustive.
College student debt - agreed, wording should be "graduates" not "students".
Animal species - the wrong source is used here, thanks for pointing that out. The source for the 30-50% answer should be http://www.biologicaldiversity.org/programs/biodiversity/ele..., which refers to this Nature article: http://www.nature.com/nature/journal/v427/n6970/full/nature0...
Illiteracy - my understanding of functional illiteracy is in line with how Wikipedia defines it, and I think it's a reasonable phrasing: https://en.wikipedia.org/wiki/Functional_illiteracy
Mental health in prisons - this feels a little like splitting hairs here, but the US Dept. of Justice source defines "having mental problems" as "a recent history or symptoms of a mental health problem...A recent history of mental health problems included a clinical diagnosis or treatment by a mental health professional. Symptoms of a mental disorder were based on criteria specified in the Diagnostic and Statistical Manual of Mental Disorders, fourth edition". So the wording of the question seems reasonable to me, in accordance with the Dept. of Justice (and including diagnosis by a professional).
On a larger scale, I agree the project is of course biased, both in terms of the questions selected and the multiple choice answers, as is stated right on the site. And I don't know if that is a bad thing. As is mentioned, the point of the site is to highlight things that are consistently misrepresented in the media, and to compare actual data to the common misconceptions that result from this misrepresentation in the media. To do that, the answer bins do need to involve what we believe to be these misconceptions as options.
Above all though, the point is to motivate discussion around these questions, and hopefully this project can help do that. Thanks again for your comment, we will fix those mistakes I mentioned above right away. Cheers!