Sunday, June 18, 2006

Lying with Statistics: Today's Example

CNN: "Report: Fatalities soar after helmet law lifted"

Clicking on the link, one discovers that:

"Motorcycle fatalities involving riders without helmets have soared in the nearly six years since Gov. Jeb Bush repealed the state's mandatory helmet law, a newspaper reported Sunday.

A Florida Today analysis of federal motorcycle crash statistics found "unhelmeted" deaths in Florida rose from 22 in 1998 and 1999, the years before the helmet law repeal, to 250 in 2004, the most recent year of available data.

Total motorcycle deaths in the state have increased 67 percent, from 259 in 2000 to 432 in 2004, according to National Highway Traffic Safety Administration statistics.

Records, though, also show motorcycle registrations have increased 87 percent in Florida since Bush signed the helmet law repeal July 1, 2000.


Deaths went up 67%, registrations went up 87%, so deaths per motorcycle have been going down. "Unhelmeted deaths" went up steeply, which sounds convincing—until you realize that one result of not wearing a helmet is that an accident that would have killed you even with a helmet now counts as an "unhelmeted" instead of a "helmeted" death. I do not know what else changed over the period; it would be interesting so see comparable statistics from states that did not change their laws. But the evidence actually presented in the article, taken by itself, implies precisely the opposite of what the top level headline suggests.

24 Comments:

At 11:13 PM, June 18, 2006, Anonymous Anonymous said...

I've noticed:
If you sound authoritative (news analyst, pundit, politician, doctor) and make a point supported by something people don't know much about (statistics, biomedical "research studies"), people seem to take your statements to heart without being very critical.

Alcibiades

 
At 11:41 PM, June 18, 2006, Anonymous Anonymous said...

Professor,I am a teacher from China. I was learning from your articles and books, and now including your blog. My blog is http://laws.fyfz.cn/blog/laws/
Maybe I could communicate with you though email some time.
danhon@163.com

 
At 12:44 AM, June 19, 2006, Anonymous Anonymous said...

I'd like to know why metastasizing government hasn't yet decided to ban motorcycles completely. After all, there are seatbelt laws, and one can't wear a seatbelt on a motorcycle, can they?

 
At 6:26 AM, June 19, 2006, Anonymous Anonymous said...

But then, to play devil's advocate:

Is the number of motorcycle registrations a good proxy for the "expected" level of accidents?

Is the surge in registrations coming from novice motorcyclists who (like anyone else new to a dangerous activity) are at higher risk of getting in more trouble than they can cope with?

Or are they coming from experienced motorcyclists migrating to FL because of the lack-of-helmet law?

Are motorcyclists who rode but were unregistered before now registering because they can now ride legally?

Any data on what fraction of accident victims were wearing helmets before and after the law change?

 
At 6:37 AM, June 19, 2006, Anonymous Joshua Kronengold said...

It's possible, (as one of the anonymous respondents mentioned) that the 87% increase in registrations isn't an indicator of 87% more riders on the road, but of, say, the weakened laws making it more likely for people to register their motorcycles. One would need more data to determine this, obviously.

 
At 8:09 AM, June 19, 2006, Anonymous Tom Courtney said...

Deaths went up 67%, registrations went up 87%, so deaths per motorcycle have been going down.

That's not what I took from the article. I took the "registrations since the law was repealed" bit to mean new registrations, not total. If that's true, and the extant base is large enough compared to the new registrations, than the new registrations going up can still mean that deaths/motorcycle is going up.

I'm hoping that once people start understanding the implication of the web, they'll post links to their data in their articles.

 
At 8:34 AM, June 19, 2006, Blogger SheetWise said...

Surely there are riders whose decision to wear a helmet is based on the circumstances. Passenger miles for each group, helmeted and bald, new and old registration, urban/rural, young/old, experienced/inexperienced, commuters/pleasure riders etc.

I don't see how any conclusions can be drawn from the data given. I find it interesting that the reporter used bias to see something that wasn't there -- rather than constructively using their bias to conclude that the data is counterintuitive. Bias isn't necessarily a bad thing -- in this case it could lead the reporter to honestly conclude that they need more data.

 
At 12:47 PM, June 19, 2006, Blogger David Friedman said...

Tom "took the "registrations since the law was repealed" bit to mean new registrations, not total."

I suppose that's possible, but I don't think it is the natural way of interpreting the statement. That would be something like "the rate of new registrations." I expect all motorcycles in Florida are legally required to be registered, so registration is a measure of the total number of motorcycles out there.

Of course, number of miles driven might not increase at the same rate as number of registrations; as I tried to suggest, there are a variety of complications that prevent a confident conclusion from the data. But I still think that, insofar as the data reported tell us anything, it is the opposite of what the top level headline implied.

 
At 3:49 AM, June 20, 2006, Blogger Tim Lambert said...

NCSA stats for Florida estimate that in 2004 there would have been 94 fewer motorcycle deaths if helmet wearing was 100%. Given that deaths increased by 173 and helmet wearing was probably only 90% or so before the law changed, that means that only half of the increase in deaths can be attributed to the repeal of the helmet law.

Other data at the NCSA site shows that while national motorcycle registrations increased 25%, the distance travelled on motorcycles declined slightly, so the registration figures given in the article aren't a good way to control for increased motorcycle use.

So the headline was sort of OK and the real problem was including registration numbers rather than distance travelled in the body of the article.

 
At 8:21 AM, June 20, 2006, Anonymous albatross said...

tim lambert:

Well, the headline was okay if the reporter or editor was aware of the additional information you cited. If they just looked at the data they quoted, it looks like either the reporter or the editor decided on a headline that looked plausible to him, without carefully thinking through what the statistics meant.

Now, it would seem like a really surprising result if helmet laws didn't decrease motorcycle fatalities (assuming you have any noticeable rise in helmet use, and that helmets have any noticeable effect on decreasing fatalities). But it's really good to at least look at what the numbers say. And if the reporter was familiar with the numbers you cited, he should have mentioned them.

 
At 11:14 AM, June 20, 2006, Blogger David Friedman said...

This comment has been removed by a blog administrator.

 
At 11:18 AM, June 20, 2006, Blogger David Friedman said...

Tim Lambert writes:

"NCSA stats for Florida estimate that in 2004 there would have been 94 fewer motorcycle deaths if helmet wearing was 100%. "

Perhaps I missed it, but I didn't see any explanation of how that estimate was created, which makes it very nearly worthless—mere assertion.

The obvious way is by comparing the ratio of deaths to accidents for those with and without helmets. But that assumes that wearing a helmet doesn't affect the probability of an accident, which isn't all that plausible an assumption.

As you probably know, Peltzman's old piece on auto safety legislation provided evidence that that mistake was responsible for all of the claimed reduction in auto death rates due to the legislation.

Off hand, I can't think of any way NCSA could have generated the figure that doesn't run into some problems of that sort. Even if they had accidents per mile for those who did and did not wear helmets—which seems unlikely—it would still be possible, even likely, that helmet wearing correlated with other characteristics that affected the accident rate.

Albatross thinks it would be "a really surprising result if helmet laws didn't decrease motorcycle fatalities." I disagree.

Two reasons it might not are that helmets make it harder for the rider to see and hear, and that riders may be more careful when not wearing helmets because they know they are more vulnerable.

 
At 2:10 PM, June 20, 2006, Blogger Glen Whitman said...

I'm not sure if the figures are for the same time periods. If you look at this page, you can see the raw numbers. It includes the number of registrations for 1994-2002, and the number of motorcycle fatalities for 1994-2003. When I calculate the number of fatalities per registration for 1994-2002, the seems to be an upward trend, with the highest rates in 2000, 2001, and 2002. I could not replicate the 87% increase in registrations figure.

 
At 6:15 AM, June 21, 2006, Blogger Tim Lambert said...

The NCSA calculations of helmet effectiveness are here.

They looked at death rates in accidents with two people on the same bike where one was wearing a helmet and the other was not. This controls for accident severity.

Other evidence is that the helmetless are overrepresented in deaths as compared to their percentage of riders. (This is measured by observing usage among riders, so controls for the possibility that one group rides more.)

Other researchers have done better analyses (eg looking at cars with and without such features separately) of the same safety features that Peltzman looked at and found that they did save lives, and I don't think that researchers in the field believe that such risk compensation is common.

 
At 1:28 PM, June 21, 2006, Blogger David Friedman said...

My thanks to Tim Lambert for posting a link to the helmet information. I find it interesting that the different results they mention are, on the face of it, internally inconsistent. They are the results of different studies in different places, so that doesn't imply any dishonesty, but it does suggest that any particular result should be viewed with some skepticism.

Specifically, the piece concludes that helmets used to be 29% and are now 37% effective in preventing death in a crash. But it cites a study finding that unhemeted riders were 3.4 times more likely to die in a crash and a study in Colorado that found a 100% increase in fatal crashes when helmet used declined from 99% to "as low as 49%."

Unless the effectiveness of helmets varies radically from place to place, it is hard to see how those claims can all be true. And all of them are being cited in support of arguments for making helmet wearing mandatory.

The study Tim links to provides strong evidence that, if you are in a crash, you are better off wearing a helmet, but I don't think it provides any evidence at all--perhaps I missed it--of how wearing a helmet affects the probability of being in a crash.

 
At 7:53 PM, June 21, 2006, Anonymous albatross said...

I hadn't thought about the risk homeostasis issue--you're right, maybe it wouldn't be so surprising if wearing helmets didn't decrease deaths. One reason to think they might is that, as far as I understand, it's less pleasant to ride with a helmet than without. So I think motorcycle riders (assuming no helmet law) make a tradeoff between safety and fun. That's a legitimate tradeoff, but I assume it does have a cost as well as a benefit.

 
At 6:09 AM, June 26, 2006, Anonymous Anonymous said...

Let's see...
Total deaths 1999 = 259 (thereabouts) of which 22 were unhelmeted. That means 237 deaths were with helmets
Total deaths 2004 = 432 of which 250 were unhelmeted. That means 182 deaths were with helmets.

So, deaths of unhelmeted have RISEN by 228 (call it 1000%) and deaths of helmeted have DROPPED by 55 (call it 25%) despite an increase in registrations of 87%.

Natural selection in action.

Cheers!

 
At 5:00 PM, July 08, 2006, Anonymous Latarnik said...

Some people working at the morgue tell me that motorcyclists who died in an accident wearing helmets have beautiful faces and broken spines. In France they sell long steel cable to protect motorcycle from being stolen when parked and protect rider from unnecessary suffering when put on the neck line a noose. When at the high speed, vehicle and rider go different ways, cable cuts the head off very quickly, no long sickness.

 
At 12:33 PM, August 03, 2006, Anonymous Anonymous said...

An economics blog talking about lying with statistics is quite amusing.

Where are the statistics on how much American consumers lose on depreciation of automobiles every year? There aer 200,000,000 cars in the US. Cars wear out with use and therefore deperciate. What is the total?

But if you check an economics book depreciation is defined as applying to capital goods, therefore all of the consumer automobiles don't depreciate. Oh, my mistake! That is not lying with statistics that is lying with definitions.

http://discussions.pbs.org/viewtopic.pbs?t=28529

psikeyhackr

 
At 10:27 AM, August 13, 2006, Blogger Dr. T said...

People also like to talk about how the death rate in auto accidents has decreased since we put in seat belt laws. Of course, they fail to mention that one way the rate could go down is if the same number of people die, but the number of accidents go up (due to people being less careful, since they know that everyone else is wearing their seatbelts, so everyone else is "safe" even if they are driving unsafely).

 
At 6:51 AM, January 21, 2009, Blogger wow power leveling said...

This comment has been removed by a blog administrator.

 
At 11:24 PM, March 16, 2009, Blogger moto said...

This comment has been removed by a blog administrator.

 
At 3:13 PM, June 04, 2009, Anonymous Anonymous said...

In the period from 1994 to 1996, about half of all 50 American states plus the federal district had laws compelling all motor cyclists to wear a helmet, while the other half of jurisdictions did not. Through a comparison of the law states with the no-law states over this period, it was found that the helmet laws failed to have a significant impact on the fatality rate per 10,000 registered motorcycles. The authors mention the psychological principle of “risk compensation” as a possible explanation, and quote several earlier authors on the topic of crash helmet legislation in the US who had also referred to this principle in order to explain their respective findings. The only factor in Branas and Knudson study the that appeared to make any differences in motorcyclist death rates between the states was the length of the motorcycling season (Branas, C.C. and Knudson, M.M. (2001). Helmet laws and motorcycle rider death rates. Accident Analysis and Prevention, 33, 641-648).

 
At 3:17 PM, June 04, 2009, Anonymous Gerald JS Wilde said...

In the period from 1994 to 1996, about half of all 50 American states plus the federal district had laws compelling all motor cyclists to wear a helmet, while the other half of jurisdictions did not. Through a comparison of the law states with the no-law states over this period, it was found that the helmet laws failed to have a significant impact on the fatality rate per 10,000 registered motorcycles. The authors mention the psychological principle of “risk compensation” as a possible explanation, and quote several earlier authors on the topic of crash helmet legislation in the US who had also referred to this principle in order to explain their respective findings. The only factor in Branas and Knudson study the that appeared to make any differences in motorcyclist death rates between the states was the length of the motorcycling season (Branas, C.C. and Knudson, M.M. (2001). Helmet laws and motorcycle rider death rates. Accident Analysis and Prevention, 33, 641-648).

 

Post a Comment

Links to this post:

Create a Link

<< Home