Loaded Dice: Professor Altemeyer's response

Some time back, I had a post taking issue with research on "right wing authoritarianism" by Robert Altemeyer contained in a webbed book of his. Eventually someone called Professor Altemeyer's attention to the post and he responded in a comment. Since people are unlikely to notice an exchange in the comment section of an old post, I'm shifting the discussion to a new one.

My central complaint was that he had first defined RWA in a fashion that purported to be politically neutral, with "right wing" having to do with attitude to established authority not with whether one voted for Republicans or Democrats, and then biased his test in a way that would consistently make individuals on the political right look more authoritarian than they were and individuals on the left look less. I don't think his response adequately deals with that complaint, but perhaps if I explain why he can show me that I am mistaken.

To quote from my previous post, describing the 20 questions on whose answers Professor Altemeyer based his measure of how right wing authoritarian the responder was:

What is almost immediately obvious if you read the questions is that they aren't testing for RWA as the author defines it but for a combination of that and right/left political views. When the question is of the form "people who campaigned for unpopular causes X, Y and Z were good," X, Y and Z just happen to be causes more popular on the left than on the right. When the question is of the form "We should follow authority X," X just happens to be a source of authority, such as the church, more popular on the right than on the left. No questions about people who campaigned for unpopular right wing causes or about deferring to sources of authority popular on the left.

Professor Altemeyer responds:

"When one is measuring submission to established authority in a society, one has to mention those authorities, their views, etc. in the items."

That, of course, is true. But it doesn't answer my objection, which is about the particular authorities you selected. As I pointed out in the longer discussion in the Usenet thread, one could easily enough replace your questions with others in which the authority was one popular with the left and unpopular on the right, or the unpopular cause one popular on the right and not the left.

Suppose, for instance, that one of the questions asked whether a worker should be willing to cross a picket line and go to work if he disagreed with the decision to call a strike. Labor unions are established authorities, so someone who disagrees is demonstrating RWA as the book defines it. But I predict that that question would have shown people on the left more RWA and people on the right less than the corresponding question you used.

Similarly, if instead of asking how the responder felt about "those who challenged the law and the majority's view by protesting for women's abortion rights, for animal rights, or to abolish school prayer" you asked about those who challenged the law and the majority view by picketing abortion clinics--abortion is, after all, legal, and has been for decades--or about those who challenged the majority view by home schooling their children in order to give them a proper religious education, you would have gotten a rather different pattern of responses.

My complaint isn't that you are not measuring authoritarianism--I'm a libertarian, and I indeed came out with a fairly low score on the test. It is that you are measuring a combination of authoritarianism and right wing political beliefs. Given the bias built into your test, if a right winger and a left winger are equally authoritarian, the right winger will get a higher score. That is a fatal fault in a test which you use to justify the claim:

"In North America people who submit to the established authorities to extraordinary degrees often turn out to be political conservatives,"

You can't justify such a claim using a test which is in part testing for political conservatism.

Professor Altemeyer in his response points me at footnote 7, which deals with ambiguity and bias in the questions. So far as I can tell, it is irrelevant to my point. If his test produces a score which is, say, .6 a measure of authoritarianism and .4 a measure of political belief, the results could be internally consistent and still produce a biased result. That would be less true if my criticism applied to only a few questions, since answers on them would correlate poorly with answers on the rest of the test. But in fact, as I point out in the Usenet thread, a majority of the twenty questions are politically biased, measure political beliefs as well as authoritarianism. He has come up with an internally consistent set of questions, all right, but they are measuring the wrong thing. Indeed, given that he was discarding questions that didn't correlate well with the rest of the test, if he had put in one of mine (see below) where the political bias was reversed, he would have concluded that it was a bad question and discarded it.

Let me try to put a series of questions to Professor Altemeyer, to see if we can identify what we disagree about:

1. Is it true that, in defining "right wing authoritarianism," you claim that you are not using "right wing" in a political sense?

2. Is it true that, in your list of questions, the authorities you choose to test submission to are consistently authorities more popular with the right than the left and the anti-authoritarian causes you test approval of are consistently ones more popular with the left than the right–so consistently that there is not a single question that goes the other way?

3. Do you agree that such a set of questions will consistently show a higher level of RWA for people on the right than for people on the left, actual degree of authoritarianism held constant?

4. Do you agree that if all the above points are correct, your results cannot justify your conclusion that people on the right are more authoritarian than people on the left?

For your entertainment, here is a list of alternative questions that one would use to replace some of yours if one wished to reverse the political bias; it's from one of my usenet posts in the thread on this subject.

23: When a union calls a strike, workers should decide for themselves whether it is justified and cross picket lines to go to work if they think it is not.

24. Our country desperately needs a decisive leader who will overcome special interest politics and break the political power of big corporations in order to do what is good for the common people.

25. Fundamentalist Christians are just as healthy and moral as anybody else.

(Incidentally, the original of that, with "gays and lesbians," is another question where someone who actually thinks about it clearly will give just the opposite of the pattern the author assumes. Gays are not just as healthy as the rest of us--they have a much higher rate of AIDS. So "strongly agree" on that question means "say the politically correct thing even when I know perfectly well it is false." Which sounds like authoritarianism.)

26. It is always better to trust the consensus of the scientific community on issues such as global warming, rather than to listen to the ignorant sceptics in our society who are trying to create doubt in people's minds.

27. You have to admire those who challenged the law and the majority's view by pushing for the abolition of affirmative action, for laws allowing ordinary citizens to carry firearms for self defense, for school voucher programs to let parents get their kids out of the trap of failing public schools.

A secondary objection that I offered to the list of 20 questions was that on two of them, one mentioned in my initial post, another in the Usenet discussion, the answer of a thoughtful respondent would go the wrong way–the non-authoritarian would give what is supposed to be the authoritarian answer. In each case, the reason is that the "non-authoritarian" answer is wrong. We have no reason to believe that atheists are "every bit as" virtuous as church goers–they might be more virtuous, they might be less. We have very good reasons to believe that gays are not as healthy as non-gays, given the existence of AIDS.

Professor Altemeyer responds:

"Is someone who strongly agrees with Item 6 showing authoritarian submission to a sub-group of skeptics? Possibly. But I doubt it. Atheists and agnostics have a pretty strong streak of individualism running through them--which is one of the reasons they are non-believers in a believing society."

The question isn't whether atheists are authoritarians. The question is whether some version of the idea "you aren't supposed to say that some groups of people are better than others" is common in our society. The answer is that it is, and it is the only reason I can see why someone who actually thought about those two questions would give what is supposed to be the "non-authoritarian" answers to them. Would you agree that someone who gives an answer he knows is wrong in conformity to that sort of social pressure is demonstrating what you call "right wing authoritarianism?"

Finally, let me thank Professor Altemeyer for his courteous response to my post. He didn't even complain that I should have told him I was criticizing him online–although if he had I would have responded that I have so far been unable to locate an email address for him, and snailmail and the telephone are so 20th century. My email address, in case he wants to shift part of the discussion to email, is ddfr@daviddfriedman.com. But I hope he will also respond here.

Blog to Book, an Open Source Project

In a recent post I raised the possibility of producing a book based on the contents of a blog such as this. My initial feeling was that the project wasn't worth doing, since the material was already available to be searched and read in its present form. A number of comments offered persuasive arguments on the other side, suggesting that, for at least some blogs, the material could be made considerably more useful and accessible by the sort of selection and organization that would accompany the conversion into a book.

At which point it occurred to me that, while the project might indeed be useful, there was no particular reason why I—more generally, the blog author—had to be the one who did it, or even had to give permission for someone else to do it. 

Initially I was imagining that what was being produced would be a physical book, a hardcopy, but that is arguably an obsolescent technology anyway. Suppose instead what is produced is a web page, a hyptertext table of contents to a blog, whose purpose is to select out and organize those parts of the blog of interest to the author of the page. Each entry links to the corresponding blog post; notes clarify what is where. It is not a full substitute for the original proposal, since the author of the web page, unlike the author of the blog, is not in a position to combine three posts into two, eliminating duplication, or revise the post that started a discussion to take account of what came later. But it could provide quite a lot of the additional value that would be provided by the earlier version. In particular, it could let someone interested in my political ideas follow that subset of the blog without being distracted by my search for the perfect pocket computer/internet device, and it would make it easier to see the connections between my views of how to organize the world and my views of how to bring up children.

At which point it also occurred to me that the project itself need not have a single author. It could be in the form of a Wiki targeted at the blog. There could be multiple such web pages, written by authors interested in different subsets of the blog content. There could be dueling versions, one by a fan of my political philosophy, one by a critic, each using his organization of and comments on my posts to support his view.

In at least one important way, this would be an easier project than my original version. A significant amount of the material on my blog was written by other people. That's obviously true of the comments. But I also had extended exchanges with two other professors, Robert Frank (who I managed to confuse with Robert Ellickson in my original post on this subject), and Robert Altemeyer, who came on the blog to defend his work against my criticism. Their contributions are, morally and I suspect legally, their intellectual property, not mine, so I could not legitimately include them in a book I authored without their permission. Very possibly they would give it, possibly not—and if we imagine stretching such a project far enough into the future, some of the people whose permission was required might no longer be around to give it. I could, of course, give my summary of their arguments, but that would be less informative and, I think, less convincing, than their version. But given that their contributions are already available online, no permission is required to link to them.

At which point this is becoming not so much an idea for my blog as a speculation about the ways in which exchanges of ideas and arguments might evolve over the next few decades, given the technologies now available.

More Loaded Dice

Several years ago I had an exchange on this blog with Professor Robert Altemeyer over his claim that authoritarianism was more common on the political right than on the political left. I argued that the survey on which his claim was based was, probably not intentionally, loaded. Questions about respect for authority consistently referred to authorities more popular on the right than the left, questions about people bravely defying authority referred to forms of defiance more popular on the left than on the right, hence people on the left would appear, by their score on his questions, less authoritarian than they were, people on the right more. I recently encountered the same problem in a different context, this time an article describing a study that purported to show that people on the right are more often misinformed about public issues than people on the left.

The obvious way to rig the results of such a poll is to select questions where the answer you consider mistaken is more popular with one side than the other. Most people who believe Obama was not born in the U.S. are on the right. Most people who believe the Chamber of Commerce used foreign money to influence the most recent election are on the left. By my count, for at least seven of the eleven questions the answer that the study's authors considered misinformed was a view more popular with the right than the left. One—the Chamber of Commerce question—went the other way.

A second problem with the study was that, for at least three of its eleven questions (whether stimulus had saved several million jobs, whether the economy was recovering, whether Obamacare increased the deficit), the right answer was unclear. In none of the three did the study's authors provide adequate support for their view—which, in each case, coincided with the claims of the Administration.

I first heard of the study via a critical piece on Reason's blog. A while later, I came across another reference to it, a Usenet post by someone who obviously approved of its conclusions. I responded and pointed out the problems.

With regard to the three questions where the study's answer was less obviously correct than its authors thought, I can easily imagine a reasonable person disagreeing with me, arguing that the study at worse mildly exaggerated how clear the right answer was. I do not, however, see how any reasonable person could fail to see the way in which the selection of questions was biased, once it was pointed out.

I am now waiting to see if there is anyone reading that particular Usenet thread who is willing to admit that the evidence for a conclusion he likes is bogus.