Evaluating Controversial Claims

Someone makes a scientific claim, typically as an argument for some policy. Examples would be current claims with regard to global warming, claims fifty years ago about the consequences of population growth, claims early in the Obama administration about the need for large deficits to bring down unemployment. There are at least four different ways in which an interested observer can decide whether or not to believe the claim.

1. Partisanship. If you support the policy, believe the claim. If you don't, don't. This is probably the most common approach.

2. Evaluate the arguments for yourself. This is  the most entertaining and educational approach but no more reliable than the first—and likely to give the same answer. There is always controversy about the claim among people better equipped to evaluate it than the random observer, although one side or both may try to deny it. In the case of global warming, the relevant claim is not merely that temperatures are going up, or that the reason is human activity, or that they can be expected to go up by enough to cause serious net costs, but all of those plus the additional claim that there are ways of reducing the increase that are worth their cost. To evaluate all of that you need a reasonably expert knowledge of climatology, statistics, ecology, economics, and probably two or three other fields I have not thought of. Since you don't have all that, you end up believing whichever arguments you want to believe.

3. The argument from authority. You try to figure out what the consensus of the people who are experts is or what some authoritative source of information says. An outsider trying to figure out what professionals in a field believe is at risk of overvaluing whatever position has the most support from public sources of information, such as the mass media, or has done the best job of getting its supporters onto the committees of scientific organizations that put out public statements. And even if he could figure it out for one field, that isn't sufficient. Again taking the global warming case, it is not enough to know what the consensus of the climatologists is, even if you can separate the facts from the puffery on that subject. Climatologists are not economists, so  could be correct about the expected temperature increase and wrong about the magnitude or even the sign of its consequences. Economists are not ecologists, so might show the costs they are looking at to be insignificant while missing the effects of climate change on other species. I discussed problems with this approach at greater length in an earlier post.

4. Prediction. Once such a controversy has been going for a while, partisans have a track record. If they have made confident predictions that turned out to be wrong, that is good evidence that they are either dishonest or arguing from an incorrect theory. Figuring out whether the arguments for a theory are right or wrong is much harder than finding out what that theory predicted. Sometimes all the latter takes is  a book or article by its supporters written a few years back.

The clearest case is the population hysteria of the 1960's. Paul Ehrlich's Population Bomb, published in 1968, confidently predicted mass famine in the third world over the next decade, with hundreds of millions of people starving to death. Not only did it not happen, the real world moved in the opposite direction, with calorie consumption per capita in the third world going up, not down. That is very strong evidence that Ehrlich can not be trusted. It is somewhat weaker evidence that the movement of which he was part, whose members generally took him and his arguments seriously, can not be trusted.

Weaker examples apply to my other two cases. Early in Obama's first term, the Administration offered predictions of what the unemployment rate would be without a stimulus and how much lower it would be with the stimulus that the Administration wanted and got. Actual unemployment rates for the next several years were higher with the stimulus than predicted without it. That does not tell us whether the stimulus was a good policy. It is possible, as its supporters argued after the fact, that things were simply worse than they thought. But it is good evidence that predictions made by Administration economists can not be trusted, that either they were deliberately fudging the results or were using models much less reliable than they claimed.

For the case of global warming, we have the IPCC's repeated overpredictions of global temperatures, hurricane rates that are strikingly lower, not strikingly higher, this year than the average, and a number of other predictions to which the real world has failed to conform. Again, that does not show that the underlying argument is wrong. It does show that the people and models that have been generating the mistaken predictions cannot be trusted.

Which is about the most that the interested outside observer can hope to learn.

---

P.S. Not surprisingly, a lot of the comments on this post focus on the particular case of global warming rather than the general argument. One commenter provides a link to an article by Chris Landsea on the effect of global warming on the frequency and strength of hurricanes. One conclusion is that global warming by the end of the century might result in a slight increase in the strength of hurricanes and might also result in a substantial decrease in their frequency. 

Another is that the historical record provides no support for the claim that hurricanes have been becoming more frequent or stronger over the past century or so. The reasons that hurricane damage has trended up is that the total value of property in coastal counties has increased. The reason that the number of recorded hurricanes has trended up is probably the large improvement in our ability to detect hurricanes, in particular ones that fail to make land. For details see the article.