Loaded Dice Continued
A commenter on the previous post writes:
“I would suggest that it's more likely that the questions were chosen because they were the most significant and common misunderstanding raised and that the fact that they are more common among rightists is not surprising in light of the literature suggesting that such misconceptions are more common among rightists.
Indeed, trying to select questions with the intention of "balancing" left-leaning and right-leaning errors rather than selecting questions impartially would really be loading the dice.”
Seen from one angle, the question is evidence of the problem pointed out in the biblical phrase about motes and beams. It apparently didn’t occur to the commenter that his view of what errors which people make might be affected by his own beliefs. I have, after all, just offered a sample of “the literature” on that subject as written by people on one side of the political spectrum and explained how they can get their result whether or not it is true.
But from another angle, it raises a legitimate question. If one side makes more errors, then isn’t an “unbiased” set of question actually biased? If so, how could one control for that problem?
The best solution that occurs to me is to test not for number of errors but for number of people who believe each error. Write a list of questions in which about equal numbers are errors popular with each side and see what fraction of people on each side subscribe to their side’s errors. There are still potential problems—you could bias the test by picking wildly implausible errors on one side and only mildly mistaken ones on the other. But at least you have eliminated the particular source of bias I discussed in my previous post.
You still have the problem of making sure that your right answers are really right, but I suspect someone who had the same biases as the authors of the survey we are discussing but was both more competent and more honest could manage it. One way would be to ask someone with the opposite biases to check over the answers and see if there were ones that he could offer legitimate arguments against.
My other response to the comment was to compile a list of questions designed to expose misinformation on the left. Here are some candidates:
According to the Congressional Budget Office, the top 1% of households pay what share of federal income tax?
Less than 10%
Between 10% and 30%
More than 30%
According to the Congressional Budget Office, the bottom 60% of households pay what share of federal income tax?
More than 20%
Between 2% and 20%
Less than 2%
After the 1929 stock market crash, Republican Herbert Hoover, during the rest of his term, responded by:
Sharply cutting government spending
Doing nothing in the expectation that the problem would cure itself
Sharply increasing government spending
The average tuition at private schools is:
Substantially lower than the average per pupil expenditure at public schools
About the same as the average per pupil expenditure at public schools
Considerably higher than the average per pupil expenditure at public schools
In the opinion of a majority of American economists, the usual effect of raising the minimum wage is:
To reduce poverty among low income workers
To increase employment by transferring income to poorer people more likely to spend it
To increase unemployment among low income workers
American K-12 teachers are paid:
Less than the average wage of American workers
More than the average of American workers but less than the average for college graduates
More than the average for college graduates
Including wages, pensions and other benefits, the average employee of the Federal Government is paid:
Substantially less than the average employee in private industry
About as well as the average employee in private industry
Much more than the average employee in private industry
Of the questions on this list, there is one which I would want to check before including it in a survey; I am pretty sure I know what the answer was a few decades back but not entirely certain that it is still the same.
Readers of this blog, even left wing readers, are a poor population for the test, since several of the questions have been discussed here. But they are invited to offer it to friends without any explanation of how it has been designed and see what the result is.