I recently came across a sentence in a reasonably well balanced article on climate issues that nicely illustrates the problem of interpreting statistical claims. The relevant passage:
Despite the relative simplicity of his model, Crowley found good agreement between the temperature fluctuations it calculated for the years 1000 AD to 1850 AD and the fluctuations actually measured from tree rings during that interval. Over that 850-year period, fluctuations in solar intensity along with volcanic eruptions could account for roughly 50 percent of the variation seen in the tree-ring record -- give or take 10 percent.
Something happened, however, after 1850. Crowley's model could only account for about 25 percent of the observed temperature changes. Something else was needed -- volcanic eruptions and solar variability were not enough.
Crowley then introduced a human-triggered greenhouse effect to the model and it produced a much better match.
....
In the case of Crowley's study, statistical tests show that the probability of his results being due to chance is less than 1 percent.
(Emphasis mine)
There are two things wrong with the final sentence. The first is linguistic ambiguity. "the probability of his results being due to chance" sounds as though it means "given the results he got, the probability that the cause was chance is less than 1 percent." What it actually means, however, is "if the results were due to chance, the probability of their occurring would be less than 1 percent."
To see the difference between the two readings of the sentence, consider a simpler case. I pull a coin out of my pocket and, without examining it, flip it twice. It comes up heads both times. My null hypothesis is that it is a fair coin, my alternative hypothesis is that it is a double headed coin.
Given the null hypothesis, the probability that the result would occur, that the coin would come up heads both times, is only .25. It does not follow that the probability that the result was due to chance is only .25 since, in my simple example, that would mean a .75 chance that the coin is double headed. To calculate the latter probability you would need to take account of the fact that double headed coins are very uncommon, hence, even after getting two heads, the odds are overwhelmingly against the coin being double headed.
Similarly for Crowley's result. It is a statement about how likely it is that his model would work as badly as it did for the period after 1850 if nothing had changed, not the probability that a change in what determined climate was responsible for its working as badly as it did.
But it doesn't even tell us that, because there is an additional assumption built into the argument—that the data from 1000 A.D. to 1850 A.D. provide a complete model of what determines climate. Suppose there is some cause of climate change that occurs rarely enough so that it did not occur in the period Crowley used to build his model, giving him no data at all on how likely it was or how large its effects. If it happened to hit around 1850, it could explain the divergence from his model thereafter, even though that divergence was entirely due to natural effects. Probability of that occurring? Unknown.
And in fact, if we accept Crowley's interpretation of his evidence, that is precisely what did happen—with "natural" interpreted to include the results of human activity. Every year for the past tens of thousands, there was a tiny probability of an industrial revolution producing enough CO2 to affect climate. It did not happen from 1000 A.D. to 1850, so did not show up in Crowley's data and the model built from that data. It did happen thereafter, producing results that look very unlikely given that model.
An earlier post on these issues, in the context of Lovejoy's similar 1% claim.
Earlier posts on the first problem in other contexts—World of Warcraft and Egyptian mummies.
12 comments:
Aren't they committing Texas Sharpshooter Fallacy?
Maybe our conceptualization of "climate change" is all wrong. Perhaps the null is that climate will change irrespective of human activity. Maybe the problem these climate scientists should be working on is whether our resource utilization is adaptable to a changing climate, rather than the faith based proposition that some alternate intervention will avert climate change.
I still don't believe any climate model that doesn't accurately model the Sun. The Sun is obviously the first order effect. The second order effect would be the kinematics of the Earth about the Sun. Not sure how any properly trained, self respecting agent of science would consider these two variables in any way constant.
I guess working on a comprehensive model is hard and probably couldn't reach politically correct answers.
With apologies if it is repetitive to say so, but tree rings have not been established as a reliable proxy to temperature. On the contrary, the size of tree rings appears to be dominated by other factors such as precipitation.
"In the case of Crowley's study, statistical tests show that the probability of his results being due to chance is less than 1 percent."
Another thing wrong with the sentence is its bad grammar. In English, one says "...the probability of his results' being due to chance...."
Nope, jimbino, you are wrong! What does the possessive have to do with anything here?
TheVidra, you must have missed the 7th grade class in sentence diagramming.
The sentence is intended to state, "Statistical tests show that, in the case of Crowley's study, the probability that the outcome was due to mere chance is less than 1 percent."
If the original sentence gained two commas to read, "In the case of Crowley's study, statistical tests show that the probability of his results, being due to chance, is less than 1 percent," the error would consist in asserting that either the probability or the results were due to chance, which is begging the question of the argument.
Read Google's entry on the use of the gerund, which is a feature of dozens of languages, including English. I personally withhold all credence when assaulted by wordy and cumbersome writing that uses such bad grammar.
jimbino, I think both phrasings are acceptable. One can say "the probability of X's being true," meaning the probability of X having the "is-true" property. Or one can say "the probability of X being true," meaning the probability that "X-is-true." It means basically the same thing; what changes is the emphasis: the first version focuses on truth as a property of X, the second places them on more equal footing (does X == true?)
No Patrick,
You, too, must have skipped Latin class and 7th-grade sentence diagramming.
Why don't you study up some and then try to diagram "the probability of X being true"? It won't work.
In fact, sensitivity to this particular solecism is what will alert you to bad science or, at least, to scientists who also missed 7th-grade diagramming at places like the Chicago Lab School.
jimbino, I don't think you're arguing your point well. I also think you're wrong as a matter of simple fact. You seem to be taking the position that "being" must be a gerund, as it is the -ing form of the verb "to be." But "being" is also the present participle of "to be."
With the gerund form, one usually uses the possessive, but with the participle form, one does not. (Though I have seen it suggested not to use the possessive+gerund form for nouns that are abstract, collective, or plural, meaning "results" should not be possessive in any case).
Gerunds act as nouns, participles as adjectives (or adverbs). So, are we to take "being true" as a thing that the results may possess, or as an aspect that results may embody? I think it at least reasonable to interpret "being true" as acting as playing the role of an adjective modifying results. Thus, "being" is a present participle, and requires that "results" not be possessive.
Thus, "the probability of results being true" is equivalent in meaning to "the probability of results that are true," or "the probability of true results."
Patrick,
I think you must mean:
The probability of "X is true (X==TRUE)" is acceptable, especially if you are a C programmer.
In any case, the formulation "the probability of X being true" is NOT acceptable.
Jimbino: I'm an educated native speaker of English. I say you are wrong. The phrase "the probability of his results being due to chance" is entirely grammatical. If you're not a native English speaker, you should now give up on this argument. If you are a native English speaker, I think you've been reading too many grammar books. Especially too many books on latin grammar - English is not a romance language.
As to the substance of the post... Perhaps humans started influencing the climate in 1850, but not through CO2 emissions, which were still negligible then.
Patrick and Unknown, here is the 7th-grade lesson in proper use of gerunds that you missed:
http://www.getitwriteonline.com/archive/022205posscasegerunds.htm
So, we can interpret the article as saying that, given two hypotheses, the probability that the null hypothesis would give results as good as a specified alternative hypothesis is less than 1%.
It appears that they mean the null hypothesis to be, "the 2-factor model explains the data after 1850."
Possibly they mean, "random variation describes the data up to 1850."
What they appear to actually say is, "If we gave 1000 monkeys computers with statistical packages, fewer than ten would produce results as good as Crowley did."
Post a Comment