Teaching Statistics with World of Warcraft
Consider the question of whether a process is actually random. Human beings have very sensitive pattern recognition software—so sensitive that it often sees patterns that are not there. There is a tradeoff, as any statistician knows, between type 1 and type 2 errors, between seeing something that isn't there and failing to see something that is. In the environment humans evolved in, there were good reasons to prefer the first sort of error to the second. Mistaking a tree branch for a lurking predator is a less costly mistake than misidentifying a lurking predator as a tree branch. One result is that gamblers routinely see patterns in random events—"hot dice," a "loose" slot machine, or the like.
Players in World of Warcraft see such patterns too. But in that case, the situation is made more complicated and more interesting by the fact that the "random" events might not be random, might be the deliberate result of programming. In the real world it is usually safe to assume that the dice which you have used in the past will continue to produce the same results, about a 1/6 chance of each of the numbers 1-6, in the future. But in the game it is always possible that the odds have changed, that the latest update increased the drop rate for the items you are questing for from one in four to one in two, even one in one. It is even possible, although not I think likely, that some mischievous programmer has introduced serial correlation into otherwise random events, that the dice really are sometimes hot and sometimes cold.
A few days ago I was on a quest which required me to acquire five copies of an item. The item was dropped by a particular sort of creature. Past experience suggested a drop rate of about one in four. I killed four creatures, got four drops, and began to wonder if something had changed.
It occurred to me that the question was one to which statistics, specifically Bayesian statistics, was applicable. Many students, indeed many people who use statistics, have a very imperfect idea of what statistical results mean, a point that recently came up in the comment thread to a post here when someone quoted the report of the IPCC explaining the meaning of its confidence results and getting it wrong. My recent experience in World of Warcraft provided a nice example of how one should go about getting the information that people mistakenly believe a confidence result provides.
The null hypothesis is that the drop rate has not changed—each creature I kill has one chance in four of dropping what I want. The alternative hypothesis is that the latest update has raised the rate to one in one. A confidence result tells us how likely it is that, if the null hypothesis is true, the evidence for the alternative hypothesis will be at least as good as it is. Elementary probability theory tells us that, if the null hypothesis is correct, the chance of getting four drops out of four is only one in 256. Hence my experiment confirms the alternative hypothesis at (better than ) the .01 level.
Does that mean that the odds that the drop rate has been raised to one in one are better than 99 to 1? That is how, in my experience, people commonly interpret such results—as when the IPCC report explained that "very high confidence represents at least a 9 out of 10 chance of being correct; high confidence represents about an 8 out of 10 chance of being correct."
It does not. 1/256 is not the probability that the drop rate has changed, it is the probability that I would get four drops out of four if it had not changed. To get from there to the probability that it had—the probability that would be relevant if, for example, I wanted to bet someone that the fifth kill would give me my final drop—I need some additional information. I need to know how likely it is, prior to my doing the experiment, that the drop rate has been changed. That prior probability, plus the result of my experiment, plus Bayes Theorem, gives me the posterior probability that I want.
Suppose we determine by reading the patch notes of past patches or by getting a Blizzard programmer drunk and interrogating him, that any particular drop rate has a one in ten thousand chance of being changed in any particular patch. The probability of getting my result via a change in the drop rate is then .0001 (the probability of the change) times 1 (the probability of the result if the changed occurred--for simplicity I am assuming that if there was a change it raised the drop rate to 1). The probability of getting it without a change by random chance is .9999 (the probability that there was no change) x 1/256 (the probability of the result if there was no change). The second number is about forty times as large as the first, so the odds that the drop rate is still the same are about forty to one.
And I suspect, although I may be mistaken, that the odds that a student who spent his spare time playing World of Warcraft would find the explanation interesting and manage to follow it are higher than if I were making the same argument in the context of an imaginary series of coin tosses, as I usually do.