Thursday, June 07, 2007

Loaded Dice: How to bias research

In a recent Usenet thread, the question was raised of whether authoritarian personality types were more likely to be on the political right than on the political left. One contributor offered a link to a webbed book by Professor Robert Altmeyer which, the poster claimed, described scientific research that showed that the right was much more authoritarian than the left. I read the beginning of the book and concluded that it was indeed interesting--as an example of how to load the dice in order to get the results you want out of supposedly objective research.

The book starts off by defining "right wing authoritarian" (RWA) in a way which purports to be politically neutral; the author offers an implausible explanation of his entirely non-political reasons for labelling it "right wing." There follows the set of twenty questions (plus two that don't get scored) used to test subjects to see how RWA they are. On each question, the responder is supposed to express a view from "strongly agree" to "strongly disagree." The grading is simple--on some questions you count as more authoritarian the more strongly you agree, on the rest you count as more authoritarian the more strongly you disagree.

What is almost immediately obvious if you read the questions is that they aren't testing for RWA as the author defines it but for a combination of that and right/left political views. When the question is of the form "people who campaigned for unpopular causes X, Y and Z were good," X, Y and Z just happen to be causes more popular on the left than on the right. When the question is of the form "We should follow authority X," X just happens to be a source of authority, such as the church, more popular on the right than on the left. No questions about people who campaigned for unpopular right wing causes or about deferring to sources of authority popular on the left.

Perhaps the worst question of all was:

6. Atheists and others who have rebelled against the established religions are no doubt every bit as good and virtuous as those who attend church regularly.

Almost nobody taking the test--my guess is literally nobody--has the data needed to know whether atheists are less virtuous, more virtuous, or just as virtuous as churchgoers. The only reason I can see why someone would respond with "strongly agree" is that he is an authoritarian accepting the authority of a particular subgroup within his society--one that is skeptical of religion. But that is supposed to be the extreme non-authoritarian answer to the question.

Interested readers can find a more detailed critique by me on usenet; the thread, including responses, is here.

14 comments:

Jonathan said...

Indeed. Regarding question 6, what a weird way to put it. I'm as non-religious as they come, but I dislike labelling myself an "atheist" and I'm not in "rebellion" against religions, I simply feel no need of them.

I think the non-religious are probably about as good as the religious, on average, but to claim that we're "no doubt every bit as good and virtuous" would be unreasonably dogmatic. There should always be some doubt about such assertions, which are unprovable even in principle.

Lippard said...

Question 6 is indeed weirdly worded.

There is some empirical evidence that nonbelievers are at least as moral as believers from comparing their prevalence in the general population to their prevalence in the prison population (they have lower prevalence in prison), but that could be biased by a tendency to falsely report religiousity by prisoners in order to obtain benefits that could include early parole. There are also Barna's studies of U.S. divorce rates by religion, which show that atheists are at the low end and Baptists are at the high end. (This supposes, however, that there is something immoral about divorce.)

One could certainly conduct surveys to ask questions about whether one has engaged in various unethical behaviors and asking about religious membership. To safeguard against the tendency to lie in one's favor, the polling could use randomized response methodology which has been used to estimate the prevalence of child molesters and how many women have had abortions. The method is that when you get to the question that calls for a negative self-report that the subject might lie about, you give them a choice--either one of two innocuous questions or the tough one--and ask them to flip a coin to determine which one to answer. You have two innocuous questions at hand, such as whether the respondent rents or owns their home, or is or is not a member of a labor union. In half of the surveys the home question gets paired with the tough question and the union question is asked separately in the same survey; in the other half they are reversed. That gives you the prevalence of that renters and union members in your sample, which should account for half of each of the corresponding answers to the tough question.

(BTW, "are you an atheist" and "do you believe in God" may themselves be sensitive questions for which randomized response would be necessary to get an accurate picture. Atheists are highly stigmatized in the U.S.)

Anonymous said...

There are also Barna's studies of U.S. divorce rates by religion, which show that atheists are at the low end and Baptists are at the high end.

You have to control these things by educational attainment and by marriage rates. I think it is likely that atheists tend to be more educated and intelligent than believers, and far less likely to be married.

The lower the marriage rate, the fewer "marginal" marriages there are, and the fewer divorces. I believe that is the reason that the south has more divorces than the northeast.

Personally, I suspect that the average atheist is probably about as moral as the average religious believer.

I call myself a "non-atheist" -- I am not "religious" per se, but I think materialism is itself a dogma and highly at variance with reality.

Anonymous said...

Political Compass

Mike Huben said...

Interestingly, the polical compass is strongly biased in a wierd way.

If you simply click "strongly disagree" for all the questions, you end up with a -4.36 on their social scale, making you strongly "libertarian". But you score a 0 on the economic scale. If you simply click "strongly agree" for all questions, you are supposedly "authoritarian".

And I'd like to know why the question on astrology affects the social scale: accepting astrology makes you less libertarian, somehow.

Anonymous said...

The whole concept of "authoritarianism" as treated in the social sciences has a dubious history, in my view, going back to the original publication of The Authoritarian Personality. The study that generated it was carried on by adherents of the Frankfurt School, a revisionist Marxist group exiled from Germany to the United States during the Nazi era. And if you look at the definition of authoritarianism they use, they quite clearly identify it with conservatism; there is no concept that support for socialism, for example, might be classed as "authoritarian."

This is rather akin to the idea of "repressive tolerance" put forth by Herbert Marcuse, another adherent of the school: opposition to income redistribution, government takeover of businesses, forced racial integration, and similar measures is considered to be destructive of freedom and progress, so allowing people to speak or write against socialist or egalitarian programs is repressive tolerance, and forcibly silencing them is liberatory intolerance. Though I may have gotten the phrasing wrong; it's a long time since I've looked at revisionist Marxism.

But in any case, that skewed concept of authoritarianism has never been fully rejected or even critiqued in the American social sciences, as far as I know. So when psychologists study "authoritarianism" there is likely an element of partisan bias and/or philosophical bias built in from the outset.

Mike Huben said...

Thanks for the leads, William. They make clear that the political compass is using the dubious f-scale of Adorno.

The wikipedia article on "authoritarian personality" does mention strong criticisms of the concept:
http://www.blackwell-synergy.com/links/doi/10.1111%2F0162-895X.00223

Abstrct:
"Adorno, Frenkel-Brunswik, Levinson, and Sanford's The Authoritarian Personality is probably the most deeply flawed work of prominence in political psychology. The methodological, procedural, and substantive errors of this study are well known, but they are frequently simply attributed to poor methodological judgments, issues of scaling (such as response set), or Freudian theories that legitimated circular interpretations. But a more fundamental bias arose from the attempt to empirically verify the existence of a "type" of person whom the researchers thought dangerous and with whom they did not empathize. This attempt involved two dangerous procedures: (1) the fusion of nominalist research procedures (in which empirical results were used to type respondents) with a realist interpretation of types (in which some people "just were" authoritarians and others not), and (2) a theoretically rich critique of the authoritarians and a lack of interest in the psychodynamics of liberals. This combination led to an intrinsically biased interpretive project that could not help but accumulate damning evidence about authoritarians. These subtler problems have haunted contemporary work in political psychology that avoids the methodological problems of Adorno et al.; Altemeyer's work on authoritarianism, which not only is free from the defects of the Adorno et al. study but also involves some methodologically exemplary experiments, is similarly distorted by asymmetries. The same fundamental problems seem to be at the heart of the weaknesses of the theory of symbolic racism to which critics have pointed. Political psychologists should regard The Authoritarian Personality as a cautionary example of bias arising from the choice of methodological assumptions."

Anonymous said...

I'm an agnotheist, but take an emotivist/Stirnerite view of morality and get a bit annoyed when atheists/agnostics try to claim they are really the moral ones. Vox Day is a rather out-there fundie, but this and this provide fairly good evidence that the compulsively misbehaving religious right is as imaginary as the morally unmoored atheists committing crimes without inhibition.

Anonymous said...

I'm glad to learn that Adorno et al.'s work has been criticized, and thanks to Mike Huber for pointing it out.

On further thought, it strikes me that there are two visible flaws in their conceptual structure, which may be aspects of the same underlying mistake:

(i) They don't have a concept of "left-wing authoritarian." Perhaps they thought that the leftist values they and everyone they knew well adhered to were universal principles, which no one who thought reflectively could have any reason to reject or oppose; perhaps they felt that imposing leftist beliefs was not authoritarian, but simple human decency. Or perhaps they weren't willing to come to terms mentally with Stalinism. I've recently exchanged comments with a present-day socialist who doesn't believe that socialism could ever be brutal or repressive; he classifies Stalin, and perhaps even Pol Pot, as "state capitalists."

(ii) They seem to identify all "right wing" beliefs as authoritarian, including beliefs in private property and the market economy—even when those beliefs appeal to voluntary cooperation, freedom of action, or resistance to state authority.

I've never been entirely satisfied with two-dimensional political grids, either, but they seem to avoid some of the usual confusions of the "left-right" spectrum.

Anonymous said...

I think some of you are confusing the classical deffinitions of conservative and liberal (wanting no change and wanting cahnge respectively) and as they are defined politically in the US today. the book makes it clear he defines right wing it means conservative in the clasical sense. an RWA would yield to the authority regardless of whether the authority is conservative or liberal in the US political sense. You are right that there would be no left wing authoratarian as defined in Altemeyer's book. Evan in the main stream press soviet hardliners were regularly refered to as right wing.

I do agree there may be some bias in the questions particularly when talking about religios authrities. I don't remember a discution about a clash of religous and state authority. Which one would the RWA back.

andy cage

Anonymous said...

You should look at Altemeyer's last published book, The Authoriarian Specter . That book has a whole chapter on Left-Wing Authoritarianism. Some of the items follow the same logic you object to here by asking about obedience to groups with obvious ideological leanings.

He spends a whole chapter talking about the scale. It's not actually useful to pull a single item out of a scale that long and critique the whole scale based on it. You need to look at the scale as a whole, its psychometric properties, and what it predicts.

There's plenty of room for criticism of his work, but it's better than "loaded dice." He's quite open about his own views, he reports what worked as well as what didn't work.

Here's an Amazon link:
http://www.amazon.com/Authoritarian-Specter-Robert-Altemeyer/dp/0674053052/ref=pd_bbs_sr_1/104-0530080-0145544?ie=UTF8&s=books&qid=1181958379&sr=1-1

Anonymous said...

The trouble with Andy Cage's statement that

I think some of you are confusing the classical deffinitions of conservative and liberal (wanting no change and wanting cahnge respectively) and as they are defined politically in the US today. the book makes it clear he defines right wing it means conservative in the clasical sense. an RWA would yield to the authority regardless of whether the authority is conservative or liberal in the US political sense.

is that it doesn't account for the situation where the existing situation is one where there is not much authority to yield to, and where the people who want change are seeking to have increased reliance on authority in their society. Its only model of change is tearing down existing authority and putting nothing in its place.

As a thought experiment, one might envision an anarchocapitalist society where some people are trying to institute a state. The people who wanted no change would presumably be "right wing" or "conservative", but calling them RWA seems odd. And the people who wanted change would be "left wing" or "liberal," but since they would want change to establish a stronger authority, wouldn't you have to call them LWA?

And this is precisely relevant to the history of American political thought, and in particular American political thought at the time when Adorno originally published, because an important strand in "conservative" thought of the time was the desire to preserve established liberty and prevent or roll back the creation of new authority—the many regulatory agencies spawned by the New Deal. Albert Jay Nock, Rose Wilder Lane, Isabel Paterson, Ayn Rand, Frank Chodorov, and Robinson Jeffers all were "right wing" because they saw the direction of change as leading toward a more authoritarian society. But if you call them "RWA" you're not getting an essential element in their thought. Though the socialists of the time found it possible to sell their program by arguing that a free market represented authority and government control represented liberation from authority. Since their time, of course, we've had more experience with "Meet the new boss—same as the old boss."

Anonymous said...

A Response from Bob Altemeyer

I'm sorry I'm so late getting to this discussion--sort of like arriving at a party three weeks after everyone has left. But I just learned of it today when someone went from this website to mine.

I'm sorry it seemed to the host that I had biased my research. I do, I think, address the issue he raised (see Note 7 to Chapter 1), and it probably would have sped up the discussion if he had dealt with those comments. But I'll try to answer here the particular criticisms he made.

When one is measuring submission to established authority in a society, one has to mention those authorities, their views, etc. in the items. (If you don't then people wonder, "Who is he talking about?" and you get a lot of mush in the answers.) Thus when the scale was used in the Soviet Union references were made to the Communist Party, not to religious leaders.

I doubt very much that a conservative who was not an authoritarian would score highly on the RWA scale(see Note 7 of chapter 6). This of course depends on what one thinks a "conservative" is. But since the most radical, authoritarian elements in the Republican Party have taken over the GOP, authoritarian sentiments have become strongly associated with "conservatism." I have higher memories, and hopes for the term. I think Barry Goldwater would score rather low on the RWA scale. I don't see him endorsing submission, aggression in the name of authorities, and suffocating conventionalism.

Item 6, which says that atheists, etc. are as virtuous as religious people, is purportedly the worst item on the RWA scale, because no one knows if they are or not. Well, that's a good point if the RWA scale were a factual test. (And someone who viewed the item that way could simply answer "0, Neutral.") But the scale is an attitude measure, and the vast majority of people have an opinion about the issue, right or wrong. (On the whole, people tend to agree with it, slightly.)

Is someone who strongly agrees with Item 6 showing authoritarian submission to a sub-group of skeptics? Possibly. But I doubt it. Atheists and agnostics have a pretty strong streak of individualism running through them--which is one of the reasons they are non-believers in a believing society. See my book, with Bruce Hunsberger, "Atheists" published in 2006 by Prometheus Press. It reports the first study of active (American) atheists ever done--which found some "warts" as well as much that was admirable. As a group, strong nonbelievers are not conformists, to anyone, and most nonbelievers show very nonauthoritarian behavior in general.

For lots of studies showing that religious fundamentalists, who would strongly DISagree with Item 6, ARE authoritarian, see Chapter 4 of "The Authoritarians."

There are studies, by the way, of whether Item 6 is factually true or not. Fundamentalists do have some mighty good points: they are charitable, happy, very supportive of their friends, and so on. They are also, however, highly self-righteous, and not nearly as morally good as they think they are. See Chapter 4; notice especially how much integrity fundamentalists and nonbelievers showed when dealing with the issue of religious instruction in public schools.

I won't try to defend the Moral Compass.

Just on an historical note, "The Authoritarian Personality" was almost immediately and severely criticized by other social scientists. In fact a whole book of
criticisms was published just four years after TAP appeared. (And NOBODY has jumped up and down on it more than I have.) (It is also a bit misleading to attribute the book to "Adorno's" thinking. He actually contributed almost nothing to the work, but was put on the research team at the end of the day by the project's sponsors. The true leader of the effort was Neville Sanford, who graciously suggested alphabetical listing [and therefore last place in the author list] as a way to settle squabbling among the other three over who would be second behind him.)

If I have left out some issues, or invited counterarguments that others think are important, I know they will respond. Even if the party is over.

David Friedman said...

Professor Altemeyer offers a long and courteous response to my post criticizing his work. Since the post is from some time back, not many people are likely to read his comment or my reply to it. I have therefore made my response in a new post on this blog, with a link to his comment above.