Saturday, August 13, 2011

Arctic Ice: Prediction That Isn't

A recent story suggests, based on computer modeling, that the decline of arctic sea ice may be a good deal further in the future than various people predicted, fifty  or sixty years instead of five or ten. It struck me because I had some posts a while back pointing out that a NASA/JPL web page was misrepresenting the facts on the subject, claiming a continued decline in the face of a (perhaps temporary) reversal. 

The other thing that struck me about the post was that the author did not understand the nature of computer modeling and how you test it:
Accuracy of future predictions was checked by running simulations of the late 20th century. The Model replicated the events of the past well enough to suggest that its forecasts of possible futures are realistic.
How are such models created? By fitting to past data. Having used that data in constructing the model, it is no longer available to test it.  Someone is said to have claimed that with ten parameters he could fit the skyline of New York. Assuming he did it, it does not follow that by keeping the same regression coefficients while increasing the range of the parameters he could predict the skyline of the rest of the country.

In order to test the predictions of a model you need to do it against actual predictions—information that didn't go into building the model.

27 Comments:

At 9:22 AM, August 13, 2011, Anonymous joeftansey said...

It is possible that they used a technique called "retrodiction", where you fit using say, all data since 1980, and see if it can predict the next 20 years.

This way, the "predicted" data and the model parameters remain temporally independent.

 
At 9:51 AM, August 13, 2011, Anonymous Anonymous said...

So, I know the answer, but where do you live?

What are you betting on climate change?

I think daddy would have a different opinion. That doesn't mean you should, but perhaps it should provide a pause,

 
At 1:16 PM, August 13, 2011, Blogger Neolibertarian said...

In order to test the predictions of a model you need to do it against actual predictions—information that didn't go into building the model.

This is a fundamental principle of parameter estimation and model fitting, which has been well studies and addressed.

Overfitting
"In order to avoid overfitting, it is necessary to use additional techniques (e.g. cross-validation, regularization, early stopping, pruning, Bayesian priors on parameters or model comparison), that can indicate when further training is not resulting in better generalization."

I suspect there's a lot more hay to make out of noting overfitting issues in economic and political models than there is in climate models.

 
At 1:17 PM, August 13, 2011, OpenID nilskp said...

It is possible that they used a technique called "retrodiction", where you fit using say, all data since 1980, and see if it can predict the next 20 years.

This way, the "predicted" data and the model parameters remain temporally independent.



Yeah, but what happens when the model fails? You update it using the data it failed to predict.

 
At 11:49 PM, August 13, 2011, Anonymous Anonymous said...

You made an assumption that the model was fitted to late 20th century data before using it to get the estimates.
It may be true, in which case you are right in that the author(s) do not understand the modelling.
Or the authors might have primed the model with pre-late 20th century data and used it to look at the late 20th century and found it good. In which case some 20th century trends might or might not have been captured correctly.
In short, you might want to talk more about that model so your thoughts come clearer.

 
At 12:45 PM, August 14, 2011, Blogger William B Swift said...

>I suspect there's a lot more hay to make out of noting overfitting issues in economic and political models than there is in climate models.

The biggest problem with climate podels is that they have become almost entirely political models.

 
At 1:28 AM, August 15, 2011, Anonymous Andy said...

The biggest problem with economic models is that they have become almost entirely political models.

Hey look, I can denigrate an entire profession too!

 
At 9:40 AM, August 15, 2011, Anonymous Anonymous said...

Many a fortune has been lost of the stock market based on this fallacy. It kind of reminds me of those casinos where the roulette wheel has a big sign showing the last twenty numbers rolled. When you are liquored up on free booze, it does seems like a great way to predict the next jackpot.

(BTW, only the former is a fair comparison, however, the silliness of it reminds me of the latter.)

 
At 11:24 AM, August 15, 2011, Blogger Neolibertarian said...

@William, your mildly clever retort does not in any way invalidate my point.

Overfitting is far less common in climate modeling than it is in economic and political modeling.

That David chooses to focus on non-existent issues with robust climate models where he has no expertise instead of actual issues with fragile economic (or political) models where he has some purported expertise, is telling.

 
At 11:42 AM, August 16, 2011, Blogger David Friedman said...

Various people point out that the researchers whose work is being described might have tested their model against data that didn't go into making it. That's entirely possible.

My comment, however, was not about the researchers but about the author of the news story I was commenting on. Nothing in the story suggested that he was aware of the problem, still less that he was aware of it and knew that the authors had dealt with it.

 
At 3:27 PM, August 16, 2011, Blogger Neolibertarian said...

Nothing in the story suggested that he was aware of the problem, still less that he was aware of it and knew that the authors had dealt with it.

I think you're confusing science reporting with peer review here.

 
At 4:32 PM, August 16, 2011, Blogger William Newman said...

ddfr wrote "In order to test the predictions of a model you need to do it against actual predictions—information that didn't go into building the model." and "Various people point out that the researchers whose work is being described might have tested their model against data that didn't go into making it. That's entirely possible."

For what it's worth, "testing" might not be the right word for it, but we can sometimes decide whether to have high confidence in a theory merely by how simple it is compared to the data. Newton's theory fit the data on planetary motion so well that even if the Earth had then been overcast for the next 600 years, we'd still be fairly confident that he had been onto something. Similarly QM is so good at predicting the locations of spectral lines that it'd be pretty convincing if somehow all the lines had already been measured before the theory was developed. Two good books developing approaches to this idea in exhaustive mathematical detail are Gruenwald _the Minimum Description Length principle_ and Vapnik _The Nature of Statistical Learning Theory_.

In practice good theories almost always succeed so well at prediction that this point isn't essential, but I think in principle it's worth remembering that overfitting can sometimes (for very concise very accurate theories) be excluded by means other than prediction.

 
At 8:02 PM, August 16, 2011, Anonymous Anonymous said...

Hm, incidentally, "The Seasteading Institute's Patri Friedman" from http://news.yahoo.com/blogs/lookout/silicon-valley-billionaire-funding-creation-artificial-libertarian-islands-140840896.html is not related by any chance? Preparing for sea level rise?

 
At 9:07 PM, August 16, 2011, Anonymous Steve B said...

my favorite quote on data fitting comes from von neumann

"With four parameters I can fit an elephant, and with five I can make him wiggle his trunk."

 
At 9:29 PM, August 21, 2011, Blogger David Friedman said...

"Preparing for sea level rise?"

I don't think the foot or two rise over a century that the IPCC calculations suggest is going to have much effect on the viability of seasteading.

It would be easier to respond to your comments if you used a name instead of posting as "anonymous." Currently I have no way of knowing whether two posts are both by you or by two different anoymous posters.

What struck me about one of the "anonymous" posts was the implicit assumption that everything was about which side you were on--that any critical comment about anything in the global warming discussion should be taken not as an argument but as a declaration of allegiance to one faction or the other, and responded to accordingly.

Have you bothered to read the IPCC reports? Any idea of the magnitude of the changes in temperature and sea level that they suggest? Do you care whether particular claims make sense or are supported by evidence? Are you bothered if "your side" makes confident claims and then revises them when they don't come true?

Or is the only interesting question "are you a good guy (believer) or bad guy (skeptic)?"

 
At 11:45 PM, August 21, 2011, Anonymous Nightrunner said...

OK, fair's fair. I will post under this username in the future. I just tried to make a joke that you/your family may be associated with the organization that would treat human-contributed climate change as a given. Sorry if you felt it was either intrusion or insult, none was intended.
Nightrunner/X-Anonymous

 
At 6:51 PM, August 22, 2011, Blogger Milhouse said...

But why would you think seasteading have anything to do with expecting the sea level to rise in the first place?

 
At 8:06 PM, August 22, 2011, Anonymous Nightrunner said...

Because there is no other reason to spend billions on it.
The juvenile ideas that greed, coercion and hate will not touch the man-made islands (as opposed to nature-made islands) are just that - juvenile.
No mass (more than a few thousand enthusiasts) sea steading will happen in the next 25 years anyways, pocket change from Peter Thiel notwithstanding.
When the governments have to resettle a few millions that will have no place to go and nothing to eat - that is when sea steading and sea (weed) farming will become inevitable.

 
At 4:25 AM, August 23, 2011, Blogger burson said...

say what?do you really think that the people behind seasteading even remotly care about raising sea levels or climate change?have you even read anything about their ideas?and wtf does greed and coercion have to do with it?also..why should people after the sea level raise resettle to the seas and farm sea weed?that makes absolutely no sense.consider just for a moment the vast territories of syberie, greenland or antarctics.get your act together.you give the impression of a solid troll

 
At 6:32 AM, August 23, 2011, Anonymous Nightrunner said...

People working on Seasteading right now are saying things their supporters want to hear. I am not talking about floating casino and bordello for intercontinental yacht set, but about the real cities - at least a few million for starters.
Greed etc has a lot to do with it - if your seasteading is worth a penny, someone will happily swim by to take that penny from you. Even if it isn't worth a penny, someone with come by to blow you up just on general/religious principles. So juvenile idea of place where everyone will just float and be nice to each other is just that - juvenile.
As for populating Siberia, how exactly do you propose to make Mother Russia to accept a few million refugees from Vietnam or Thailand? You think you are going to force them? And than you will feed them for at least the first year while they try to scratch their food from (poor) Siberian soil - at whose expense? Antarctics is even better idea. You got room to talk about trolls.

 
At 9:01 AM, August 23, 2011, Blogger burson said...

on the juvenility i will not comment as i suppose most of the readers here would disapprove with you.just a question:what is more juvenile-to try to change political course in an established country which people consider their cultural heritage and home?or try to establish a new political course on a tabula rasa like platform of a country whose succes is dependant on the voluntary decissions of it inhabitants?
as for mother russia, the better question would be why they wouldn't accept them.i mean..it could happen but it would also mean loosing a lot of revenue so im not sure they would be overly motivated to stay isolationist.also,im not even going to elaborate on the nonsense that rising seas would cause such problems or that in vietnam there would not be enough space afterwards.especially when you take into account that the raise will happen during a very long time period and the flooded areas are really not that much of a deal.see for yourself>http://flood.firetree.net/in the worst case scenario theres still antarctica and where do you see a problem there i do not know

 
At 12:04 AM, August 24, 2011, Anonymous Nightrunner said...

1. Both changes in existing society and establishing the new one are possible; neither can be realistically done without army, police, parliament, courts etc.

2. The main reason not to accept the refugees is they look and talk different and will require a lot of resources to resettle. The resources that will be needed for citizens - and the refugees aren't.

3. Antarctica with its sub-freezing mean temperatures and no soil is not so good for agriculture even if world's average goes up by 10 degrees. See?

 
At 4:04 AM, August 24, 2011, Blogger burson said...

well i agree that antarctica is a rediculous proposal.but i was merely asserting it to the possibility and your assumption, that the whole of vietnam and thailand would be flooded, which would require a tremendous rise in temp.in the previous link you can see that even a 50m rise would not cause such an armageddon.

 
At 9:26 PM, August 24, 2011, Blogger SheetWise said...

I believe that the debate is not over!

Maybe we should actually have one. Al Gore vs. ?????

Ok. That one's too easy. Who would you suggest?

 
At 11:23 PM, August 24, 2011, Blogger ibm123str said...

hot,hug,welcome to my website.thanks
Dell Latitude D410 Charger
Dell Latitude D400 Charger
Dell Latitude CP Charger
Dell Latitude D420 Charger
Dell Latitude D430 Charger
Dell Latitude D500 Charger
Dell Latitude D510 Charger
Dell Latitude D520 Charger
Dell Latitude D540 Charger
Dell Latitude D610 Charger
Dell Latitude D620 Charger
Dell Latitude D631 Charger
Dell Latitude D800 Charger
Dell Latitude D820 Charger从下往上到这里
Dell Latitude E4200 Charger
Dell Latitude E4300 Charger
Dell Latitude E5400 Charger
Dell Latitude E5500 Charger
Dell Latitude E6400 Charger
Dell Latitude X1 Charger
Dell Latitude X300 Charger
Dell Precision M20 Charger
Dell Precision M4300 Charger
Dell Precision M70 Charger
Dell Precision Workstation M60 Charger

 
At 11:23 AM, August 31, 2011, Anonymous Bert Inch said...

David Friedman Writes:

"Have you bothered to read the IPCC reports? Any idea of the magnitude of the changes in temperature and sea level that they suggest? Do you care whether particular claims make sense or are supported by evidence? Are you bothered if "your side" makes confident claims and then revises them when they don't come true?"

Nobody does this. Mankind is not composed of scientists, it is composed of primates. All primates take sides, all primates have hierarchies. You are asking this poor fellow to sublimate years and years of ingrained evolutionary programming that tells him to deal with any potential danger by FIGHTING or FLYING! The news says the world is going to end from global warming? FIGHT FIGHT FIGHT! BUY A PRIUS! Somebody tells me that my Prius isn't going to prevent global warming? He is interfering with my FIGHT! He will be indirectly responsible for the END OF THE WORLD AS WE KNOW IT! FIGHT FIGHT FIGHT!

 
At 6:01 AM, September 04, 2011, Anonymous rental mobil jakarta said...

Nice article, thanks for the information.

 

Post a Comment

<< Home