Saturday, September 12, 2009

International Health Care Comparisons: The WHO Numbers

In the course of the healthcare debate, supporters of change along the lines proposed by the administration have called attention to a World Health Organization study ranking the health care systems of 192 nations. A common claim is that the U.S., despite spending more per capita than any other country, still ranks only 37, behind most developed countries.

That version of the claim is at best misleading. There is a measure, "Overall Health System Performance," on which the U.S. ranks 37. But it is a measure that takes expenditure into account, downrating the U.S. precisely because it spends so much. The rank is 37 not in spite of the level of expenditure but because of it.

There is another measure, "Overall Goal Attainment," which does not take account of expenditure; on that the U.S. ranks 15, still behind a fair number of other countries but not nearly as many. So a more accurate claim would be that the U.S. ranks 15 despite its large expenditures.

Even that is misleading, however, because if one actually read the notes explaining how the numbers are calculated it turns out that "Goal Attainment" is based on five different characteristics of a health care system, only one of which is an (imperfect) measure of how much health care the system provides.

That one, "Health level," is average life expectancy, adjusted to make a disabled year count for less than a healthy year. It is an imperfect measure because life expectancy depends not only on health care but on lifestyle variables such as smoking or obesity and on factors such as the death rate from murders and traffic accidents. And even to the extent that it depends on health, health is not entirely a matter of health care; some environments are more unhealthy than others.

A second variable, responsiveness, measures how good people in each country think their health care system is, as determined by questionaires. On that one, interestingly enough, the U.S. comes in first—a fact that ought to worry the President. If Americans think the current system works better than any existing alternative, as they apparently do, they may not look favorably on changes to it.

The other variables all have to do with distribution. "Health distribution" purports to measure how unequal the distribution of health care in each country is. The authors wanted to use distribution of life expectancy but didn't have the data to do it. Instead they used a measure, never clearly explained, of the distribution of infant survival, apparently of how many infants die at what point in their first five years. Even for that, the relevant data existed for only a minority of countries; for the rest the report substituted an estimate based on variables such as poverty level.

"Responsiveness distribution" was calculated from questionaires and apparently designed to measure the degree to which respondents believed that various groups in their country were disadvantaged with regard to health care.

Finally, we have "fairness in financial contribution," defined as how nearly health costs are distributed in proportion to income minus the cost of food. That measure is obviously biased in favor of state run health care plans, since in order for both health care and its cost to be distributed in the way the authors of the report want there has to be a sizable redistribution of cost from poorer families getting health care to richer families paying for it.

My conclusion is that the numbers produced by the report are very nearly useless for purposes other than propaganda, since they do not provide much information on how good the health care systems of different countries are at delivering health care.

In fairness, I should add that I don't have any proposal for doing a much better job of comparing international health care systems, given the data limitations when trying to look at 192 different countries. Ideally, one would want a value added measure, something like the difference between actual life expectancy in a country and what life expectancy would be if there were no health care system at all. But I don't see any practical way of generating such numbers. One could simply use life expectancy, but that has the problems I have already described. One can try to look at particular outcomes heavily dependent on health care; the U.S. apparently does very well measured by cancer survival rates. But neither approach really tells you what you want to know.

13 comments:

FraserOrr said...

Perhaps a better estimate would be to calculate what life expectancy would be if you were to discount non health care related mortality? This would still be substantially inaccurate (not measuring lifestyle choices and environmental choices) but it would be closer. It also has the advantage that data like that is usually quite readily available. (Just as street lights are readily available to help searching for keys.)

An interesting question btw is this: to what extent is a nation's healthcare system responsible for lifestyle choices? Should one include "smoking prevention" as part of the whole healthcare system.

VangelV said...

My conclusion is that the numbers produced by the report are very nearly useless for purposes other than propaganda, since they do not provide much information on how good the health care systems of different countries are at delivering health care.

Sadly, you are totally right. Studies like the one you cite are useful for political purposes only and in large part depend on the bias of the researchers rather than actual performance of the systems that are supposedly being studied.

My dad just received a test that claims that his PSA is 178. While I am no doctor I know enough to guess that if the test results were accurate it means that he has prostate cancer that has spread. What bothers me is why his doctor had not ordered the same test for the past three years or why it will take a week to schedule a biopsy and a further week to see a specialist. If my dad had lived in the US, Thailand, Hong Kong, or China rather than Canada we would have known something definitive if a day or two rather than having to wait nearly a month for all the test to be done.

One of my wife's friends just left Canada for a few weeks in China so that she can get treatment on a back condition that has pretty much immobilized her. The medical system here has prescribed drugs and will not be able to complete all of the tests that are required for six weeks. She would rather get the necessary work done in China and be recovered in six weeks so that she can get back to earning a living once again. Yet, Canada is ranked higher than the US.

jenizaro said...

That the US comes first in responsiveness doesn't mean they think the current system works better than any existing alternative. It just means that, using the questionnaire's criteria, Americans' perception of their system is better than other countries' citizens. These two are NOT the same thing.

David Friedman said...

Jenizero is correct--I wrote imprecisely. But I think the implications for Obama are still pretty much the same.

Anonymous said...

Cancer survival rates seems to be on the right track. Looking at various health care systems as a potential customer, the question I want answered is, "if I get sick, will the system cure me?" I want to know probability of a cure, length of time to cure, etc. If the disease is incurable, then I want to know how long my life can be extended, and the effectiveness of palliative care.

Tim Worstall said...

Glenn Whitman has done a lot of work on this. Worth making contact with him. (His blog is Agoraphilia which will give you a start in finding him).

I know he got enraged about the numbers, did a lot of research and then published a paper on it.

Blake Riley said...

According to Robert Ohsfeldt and John Schneider, the US ranks 14th in overall life expectancy at birth. Once adjusted for non-health-related fatalities, the US ranks 1st. The relevant charts from their book are here.

I'd still like to see a chart of expected QALYs, adjusted in the same manner.

Anonymous said...

One of the bizzare features of the U.S. "health care debate" is the general assumption that spending money on health care is bad, unlike spending money on anything else. People don't only complain about how much the government spends, they complain about how much individuals spend. It is as if we are puppets of an all powerful health care dictatorship which is forcing us to accept treatments we don't want, or something. It's never spelled out exactly what is wrong with spending money on health care. Nobody worries about how much people spend on other things.

Tim Lambert said...

While the US has better cancer survival rates, the cancer death rates do not differ by much. The reason for the difference in survival rates is therefore not better treatment, but more detection.

Also, the adjustment for non-health related deaths doesn't seem correct. The reason it makes a difference is the much higher homicide rate in the US. The adjustment assumes that homicide victims are just like the general population, which is not true.

Anonymous said...

"The reason for the difference in survival rates is therefore not better treatment, but more detection."

If so, that's still a plus for American health, because detection is a key responsibility of health care.

"Nobody worries about how much people spend on other things."

People do complain about prices, which is a factor in how much they spend. Back when gasoline was $4/gallon, people were feeling the pinch and not entirely happy with the situation. Obviously there's not much that can be done about a fundamental problem of supply and demand, but the American medical system is so messed up (by government intervention) that it is not unreasonable to imagine there are many ways in which the same level of service can be made less costly. Of course, the solution being contemplated nowadays is precisely the opposite of what would actually work, so it's a disaster in the making. People will be screaming once America has copied Massachusetts.

IlĂ­on said...

And, of course, many governments don't count premie births (especially when they intend to let the infants die) as live births, but rather as stillbirths. This, of course, has the effect of lowering infant mortality rates as compared to the US, which does count premies as live births.

William B Swift said...

Tim,

Cancer survival rates plus cancer death rates HAVE TO equal 1. Unless you are saying other countries are counting cancer deaths as other deaths or dying of other things before the cancer kills them, which definitely happens but I would think not often enough to make a significant difference in the statistics.

VangelV said...

Cancer survival rates plus cancer death rates HAVE TO equal 1. Unless you are saying other countries are counting cancer deaths as other deaths or dying of other things before the cancer kills them, which definitely happens but I would think not often enough to make a significant difference in the statistics.

I have trouble with some of the statistic issues. The best way to illustrate it is to give an example.

Suppose you have Tom, who is 55 years old. He gets a test and finds out that he has prostate cancer. He gets treatments and survives until 68. The way the data is collected, that makes him a cancer survivor because he lived more than 5 years.

Tom's identical twin Harry does not get tested until he is 65 years old. He also gets treatments and also manages to survive until he turns 68. Harry is considered a cancer victim even though he had the cancer for as long as Tom did.

To produce valid statistics one needs to collect data consistently and to do the analysis properly. Sadly, most health care systems are heavily controlled by governments so there is little incentive for a way to evaluate ral performance.