More Loaded Dice

Several years ago I had an exchange on this blog with Professor Robert Altemeyer over his claim that authoritarianism was more common on the political right than on the political left. I argued that the survey on which his claim was based was, probably not intentionally, loaded. Questions about respect for authority consistently referred to authorities more popular on the right than the left, questions about people bravely defying authority referred to forms of defiance more popular on the left than on the right, hence people on the left would appear, by their score on his questions, less authoritarian than they were, people on the right more. I recently encountered the same problem in a different context, this time an article describing a study that purported to show that people on the right are more often misinformed about public issues than people on the left.

The obvious way to rig the results of such a poll is to select questions where the answer you consider mistaken is more popular with one side than the other. Most people who believe Obama was not born in the U.S. are on the right. Most people who believe the Chamber of Commerce used foreign money to influence the most recent election are on the left. By my count, for at least seven of the eleven questions the answer that the study's authors considered misinformed was a view more popular with the right than the left. One—the Chamber of Commerce question—went the other way.

A second problem with the study was that, for at least three of its eleven questions (whether stimulus had saved several million jobs, whether the economy was recovering, whether Obamacare increased the deficit), the right answer was unclear. In none of the three did the study's authors provide adequate support for their view—which, in each case, coincided with the claims of the Administration.

I first heard of the study via a critical piece on Reason's blog. A while later, I came across another reference to it, a Usenet post by someone who obviously approved of its conclusions. I responded and pointed out the problems.

With regard to the three questions where the study's answer was less obviously correct than its authors thought, I can easily imagine a reasonable person disagreeing with me, arguing that the study at worse mildly exaggerated how clear the right answer was. I do not, however, see how any reasonable person could fail to see the way in which the selection of questions was biased, once it was pointed out.

I am now waiting to see if there is anyone reading that particular Usenet thread who is willing to admit that the evidence for a conclusion he likes is bogus.

Loaded Dice: Professor Altemeyer's response

Some time back, I had a post taking issue with research on "right wing authoritarianism" by Robert Altemeyer contained in a webbed book of his. Eventually someone called Professor Altemeyer's attention to the post and he responded in a comment. Since people are unlikely to notice an exchange in the comment section of an old post, I'm shifting the discussion to a new one.

My central complaint was that he had first defined RWA in a fashion that purported to be politically neutral, with "right wing" having to do with attitude to established authority not with whether one voted for Republicans or Democrats, and then biased his test in a way that would consistently make individuals on the political right look more authoritarian than they were and individuals on the left look less. I don't think his response adequately deals with that complaint, but perhaps if I explain why he can show me that I am mistaken.

To quote from my previous post, describing the 20 questions on whose answers Professor Altemeyer based his measure of how right wing authoritarian the responder was:

What is almost immediately obvious if you read the questions is that they aren't testing for RWA as the author defines it but for a combination of that and right/left political views. When the question is of the form "people who campaigned for unpopular causes X, Y and Z were good," X, Y and Z just happen to be causes more popular on the left than on the right. When the question is of the form "We should follow authority X," X just happens to be a source of authority, such as the church, more popular on the right than on the left. No questions about people who campaigned for unpopular right wing causes or about deferring to sources of authority popular on the left.

Professor Altemeyer responds:

"When one is measuring submission to established authority in a society, one has to mention those authorities, their views, etc. in the items."

That, of course, is true. But it doesn't answer my objection, which is about the particular authorities you selected. As I pointed out in the longer discussion in the Usenet thread, one could easily enough replace your questions with others in which the authority was one popular with the left and unpopular on the right, or the unpopular cause one popular on the right and not the left.

Suppose, for instance, that one of the questions asked whether a worker should be willing to cross a picket line and go to work if he disagreed with the decision to call a strike. Labor unions are established authorities, so someone who disagrees is demonstrating RWA as the book defines it. But I predict that that question would have shown people on the left more RWA and people on the right less than the corresponding question you used.

Similarly, if instead of asking how the responder felt about "those who challenged the law and the majority's view by protesting for women's abortion rights, for animal rights, or to abolish school prayer" you asked about those who challenged the law and the majority view by picketing abortion clinics--abortion is, after all, legal, and has been for decades--or about those who challenged the majority view by home schooling their children in order to give them a proper religious education, you would have gotten a rather different pattern of responses.

My complaint isn't that you are not measuring authoritarianism--I'm a libertarian, and I indeed came out with a fairly low score on the test. It is that you are measuring a combination of authoritarianism and right wing political beliefs. Given the bias built into your test, if a right winger and a left winger are equally authoritarian, the right winger will get a higher score. That is a fatal fault in a test which you use to justify the claim:

"In North America people who submit to the established authorities to extraordinary degrees often turn out to be political conservatives,"

You can't justify such a claim using a test which is in part testing for political conservatism.

Professor Altemeyer in his response points me at footnote 7, which deals with ambiguity and bias in the questions. So far as I can tell, it is irrelevant to my point. If his test produces a score which is, say, .6 a measure of authoritarianism and .4 a measure of political belief, the results could be internally consistent and still produce a biased result. That would be less true if my criticism applied to only a few questions, since answers on them would correlate poorly with answers on the rest of the test. But in fact, as I point out in the Usenet thread, a majority of the twenty questions are politically biased, measure political beliefs as well as authoritarianism. He has come up with an internally consistent set of questions, all right, but they are measuring the wrong thing. Indeed, given that he was discarding questions that didn't correlate well with the rest of the test, if he had put in one of mine (see below) where the political bias was reversed, he would have concluded that it was a bad question and discarded it.

Let me try to put a series of questions to Professor Altemeyer, to see if we can identify what we disagree about:

1. Is it true that, in defining "right wing authoritarianism," you claim that you are not using "right wing" in a political sense?

2. Is it true that, in your list of questions, the authorities you choose to test submission to are consistently authorities more popular with the right than the left and the anti-authoritarian causes you test approval of are consistently ones more popular with the left than the right–so consistently that there is not a single question that goes the other way?

3. Do you agree that such a set of questions will consistently show a higher level of RWA for people on the right than for people on the left, actual degree of authoritarianism held constant?

4. Do you agree that if all the above points are correct, your results cannot justify your conclusion that people on the right are more authoritarian than people on the left?

For your entertainment, here is a list of alternative questions that one would use to replace some of yours if one wished to reverse the political bias; it's from one of my usenet posts in the thread on this subject.

23: When a union calls a strike, workers should decide for themselves whether it is justified and cross picket lines to go to work if they think it is not.

24. Our country desperately needs a decisive leader who will overcome special interest politics and break the political power of big corporations in order to do what is good for the common people.

25. Fundamentalist Christians are just as healthy and moral as anybody else.

(Incidentally, the original of that, with "gays and lesbians," is another question where someone who actually thinks about it clearly will give just the opposite of the pattern the author assumes. Gays are not just as healthy as the rest of us--they have a much higher rate of AIDS. So "strongly agree" on that question means "say the politically correct thing even when I know perfectly well it is false." Which sounds like authoritarianism.)

26. It is always better to trust the consensus of the scientific community on issues such as global warming, rather than to listen to the ignorant sceptics in our society who are trying to create doubt in people's minds.

27. You have to admire those who challenged the law and the majority's view by pushing for the abolition of affirmative action, for laws allowing ordinary citizens to carry firearms for self defense, for school voucher programs to let parents get their kids out of the trap of failing public schools.

A secondary objection that I offered to the list of 20 questions was that on two of them, one mentioned in my initial post, another in the Usenet discussion, the answer of a thoughtful respondent would go the wrong way–the non-authoritarian would give what is supposed to be the authoritarian answer. In each case, the reason is that the "non-authoritarian" answer is wrong. We have no reason to believe that atheists are "every bit as" virtuous as church goers–they might be more virtuous, they might be less. We have very good reasons to believe that gays are not as healthy as non-gays, given the existence of AIDS.

Professor Altemeyer responds:

"Is someone who strongly agrees with Item 6 showing authoritarian submission to a sub-group of skeptics? Possibly. But I doubt it. Atheists and agnostics have a pretty strong streak of individualism running through them--which is one of the reasons they are non-believers in a believing society."

The question isn't whether atheists are authoritarians. The question is whether some version of the idea "you aren't supposed to say that some groups of people are better than others" is common in our society. The answer is that it is, and it is the only reason I can see why someone who actually thought about those two questions would give what is supposed to be the "non-authoritarian" answers to them. Would you agree that someone who gives an answer he knows is wrong in conformity to that sort of social pressure is demonstrating what you call "right wing authoritarianism?"

Finally, let me thank Professor Altemeyer for his courteous response to my post. He didn't even complain that I should have told him I was criticizing him online–although if he had I would have responded that I have so far been unable to locate an email address for him, and snailmail and the telephone are so 20th century. My email address, in case he wants to shift part of the discussion to email, is ddfr@daviddfriedman.com. But I hope he will also respond here.

Loaded Dice: How to bias research

In a recent Usenet thread, the question was raised of whether authoritarian personality types were more likely to be on the political right than on the political left. One contributor offered a link to a webbed book by Professor Robert Altmeyer which, the poster claimed, described scientific research that showed that the right was much more authoritarian than the left. I read the beginning of the book and concluded that it was indeed interesting--as an example of how to load the dice in order to get the results you want out of supposedly objective research.

The book starts off by defining "right wing authoritarian" (RWA) in a way which purports to be politically neutral; the author offers an implausible explanation of his entirely non-political reasons for labelling it "right wing." There follows the set of twenty questions (plus two that don't get scored) used to test subjects to see how RWA they are. On each question, the responder is supposed to express a view from "strongly agree" to "strongly disagree." The grading is simple--on some questions you count as more authoritarian the more strongly you agree, on the rest you count as more authoritarian the more strongly you disagree.

What is almost immediately obvious if you read the questions is that they aren't testing for RWA as the author defines it but for a combination of that and right/left political views. When the question is of the form "people who campaigned for unpopular causes X, Y and Z were good," X, Y and Z just happen to be causes more popular on the left than on the right. When the question is of the form "We should follow authority X," X just happens to be a source of authority, such as the church, more popular on the right than on the left. No questions about people who campaigned for unpopular right wing causes or about deferring to sources of authority popular on the left.

Perhaps the worst question of all was:

6. Atheists and others who have rebelled against the established religions are no doubt every bit as good and virtuous as those who attend church regularly.

Almost nobody taking the test--my guess is literally nobody--has the data needed to know whether atheists are less virtuous, more virtuous, or just as virtuous as churchgoers. The only reason I can see why someone would respond with "strongly agree" is that he is an authoritarian accepting the authority of a particular subgroup within his society--one that is skeptical of religion. But that is supposed to be the extreme non-authoritarian answer to the question.

Interested readers can find a more detailed critique by me on usenet; the thread, including responses, is here.