A recent story suggests, based on computer modeling, that the decline of arctic sea ice may be a good deal further in the future than various people predicted, fifty or sixty years instead of five or ten. It struck me because I had some posts a while back pointing out that a NASA/JPL web page was misrepresenting the facts on the subject, claiming a continued decline in the face of a (perhaps temporary) reversal.
The other thing that struck me about the post was that the author did not understand the nature of computer modeling and how you test it:
Accuracy of future predictions was checked by running simulations of the late 20th century. The Model replicated the events of the past well enough to suggest that its forecasts of possible futures are realistic.
How are such models created? By fitting to past data. Having used that data in constructing the model, it is no longer available to test it. Someone is said to have claimed that with ten parameters he could fit the skyline of New York. Assuming he did it, it does not follow that by keeping the same regression coefficients while increasing the range of the parameters he could predict the skyline of the rest of the country.
In order to test the predictions of a model you need to do it against actual predictions—information that didn't go into building the model.
27 comments:
It is possible that they used a technique called "retrodiction", where you fit using say, all data since 1980, and see if it can predict the next 20 years.
This way, the "predicted" data and the model parameters remain temporally independent.
So, I know the answer, but where do you live?
What are you betting on climate change?
I think daddy would have a different opinion. That doesn't mean you should, but perhaps it should provide a pause,
In order to test the predictions of a model you need to do it against actual predictions—information that didn't go into building the model.
This is a fundamental principle of parameter estimation and model fitting, which has been well studies and addressed.
Overfitting
"In order to avoid overfitting, it is necessary to use additional techniques (e.g. cross-validation, regularization, early stopping, pruning, Bayesian priors on parameters or model comparison), that can indicate when further training is not resulting in better generalization."
I suspect there's a lot more hay to make out of noting overfitting issues in economic and political models than there is in climate models.
It is possible that they used a technique called "retrodiction", where you fit using say, all data since 1980, and see if it can predict the next 20 years.
This way, the "predicted" data and the model parameters remain temporally independent.
Yeah, but what happens when the model fails? You update it using the data it failed to predict.
You made an assumption that the model was fitted to late 20th century data before using it to get the estimates.
It may be true, in which case you are right in that the author(s) do not understand the modelling.
Or the authors might have primed the model with pre-late 20th century data and used it to look at the late 20th century and found it good. In which case some 20th century trends might or might not have been captured correctly.
In short, you might want to talk more about that model so your thoughts come clearer.
>I suspect there's a lot more hay to make out of noting overfitting issues in economic and political models than there is in climate models.
The biggest problem with climate podels is that they have become almost entirely political models.
The biggest problem with economic models is that they have become almost entirely political models.
Hey look, I can denigrate an entire profession too!
Many a fortune has been lost of the stock market based on this fallacy. It kind of reminds me of those casinos where the roulette wheel has a big sign showing the last twenty numbers rolled. When you are liquored up on free booze, it does seems like a great way to predict the next jackpot.
(BTW, only the former is a fair comparison, however, the silliness of it reminds me of the latter.)
@William, your mildly clever retort does not in any way invalidate my point.
Overfitting is far less common in climate modeling than it is in economic and political modeling.
That David chooses to focus on non-existent issues with robust climate models where he has no expertise instead of actual issues with fragile economic (or political) models where he has some purported expertise, is telling.
Various people point out that the researchers whose work is being described might have tested their model against data that didn't go into making it. That's entirely possible.
My comment, however, was not about the researchers but about the author of the news story I was commenting on. Nothing in the story suggested that he was aware of the problem, still less that he was aware of it and knew that the authors had dealt with it.
Nothing in the story suggested that he was aware of the problem, still less that he was aware of it and knew that the authors had dealt with it.
I think you're confusing science reporting with peer review here.
ddfr wrote "In order to test the predictions of a model you need to do it against actual predictions—information that didn't go into building the model." and "Various people point out that the researchers whose work is being described might have tested their model against data that didn't go into making it. That's entirely possible."
For what it's worth, "testing" might not be the right word for it, but we can sometimes decide whether to have high confidence in a theory merely by how simple it is compared to the data. Newton's theory fit the data on planetary motion so well that even if the Earth had then been overcast for the next 600 years, we'd still be fairly confident that he had been onto something. Similarly QM is so good at predicting the locations of spectral lines that it'd be pretty convincing if somehow all the lines had already been measured before the theory was developed. Two good books developing approaches to this idea in exhaustive mathematical detail are Gruenwald _the Minimum Description Length principle_ and Vapnik _The Nature of Statistical Learning Theory_.
In practice good theories almost always succeed so well at prediction that this point isn't essential, but I think in principle it's worth remembering that overfitting can sometimes (for very concise very accurate theories) be excluded by means other than prediction.
Hm, incidentally, "The Seasteading Institute's Patri Friedman" from http://news.yahoo.com/blogs/lookout/silicon-valley-billionaire-funding-creation-artificial-libertarian-islands-140840896.html is not related by any chance? Preparing for sea level rise?
my favorite quote on data fitting comes from von neumann
"With four parameters I can fit an elephant, and with five I can make him wiggle his trunk."
"Preparing for sea level rise?"
I don't think the foot or two rise over a century that the IPCC calculations suggest is going to have much effect on the viability of seasteading.
It would be easier to respond to your comments if you used a name instead of posting as "anonymous." Currently I have no way of knowing whether two posts are both by you or by two different anoymous posters.
What struck me about one of the "anonymous" posts was the implicit assumption that everything was about which side you were on--that any critical comment about anything in the global warming discussion should be taken not as an argument but as a declaration of allegiance to one faction or the other, and responded to accordingly.
Have you bothered to read the IPCC reports? Any idea of the magnitude of the changes in temperature and sea level that they suggest? Do you care whether particular claims make sense or are supported by evidence? Are you bothered if "your side" makes confident claims and then revises them when they don't come true?
Or is the only interesting question "are you a good guy (believer) or bad guy (skeptic)?"
OK, fair's fair. I will post under this username in the future. I just tried to make a joke that you/your family may be associated with the organization that would treat human-contributed climate change as a given. Sorry if you felt it was either intrusion or insult, none was intended.
Nightrunner/X-Anonymous
But why would you think seasteading have anything to do with expecting the sea level to rise in the first place?
Because there is no other reason to spend billions on it.
The juvenile ideas that greed, coercion and hate will not touch the man-made islands (as opposed to nature-made islands) are just that - juvenile.
No mass (more than a few thousand enthusiasts) sea steading will happen in the next 25 years anyways, pocket change from Peter Thiel notwithstanding.
When the governments have to resettle a few millions that will have no place to go and nothing to eat - that is when sea steading and sea (weed) farming will become inevitable.
say what?do you really think that the people behind seasteading even remotly care about raising sea levels or climate change?have you even read anything about their ideas?and wtf does greed and coercion have to do with it?also..why should people after the sea level raise resettle to the seas and farm sea weed?that makes absolutely no sense.consider just for a moment the vast territories of syberie, greenland or antarctics.get your act together.you give the impression of a solid troll
People working on Seasteading right now are saying things their supporters want to hear. I am not talking about floating casino and bordello for intercontinental yacht set, but about the real cities - at least a few million for starters.
Greed etc has a lot to do with it - if your seasteading is worth a penny, someone will happily swim by to take that penny from you. Even if it isn't worth a penny, someone with come by to blow you up just on general/religious principles. So juvenile idea of place where everyone will just float and be nice to each other is just that - juvenile.
As for populating Siberia, how exactly do you propose to make Mother Russia to accept a few million refugees from Vietnam or Thailand? You think you are going to force them? And than you will feed them for at least the first year while they try to scratch their food from (poor) Siberian soil - at whose expense? Antarctics is even better idea. You got room to talk about trolls.
on the juvenility i will not comment as i suppose most of the readers here would disapprove with you.just a question:what is more juvenile-to try to change political course in an established country which people consider their cultural heritage and home?or try to establish a new political course on a tabula rasa like platform of a country whose succes is dependant on the voluntary decissions of it inhabitants?
as for mother russia, the better question would be why they wouldn't accept them.i mean..it could happen but it would also mean loosing a lot of revenue so im not sure they would be overly motivated to stay isolationist.also,im not even going to elaborate on the nonsense that rising seas would cause such problems or that in vietnam there would not be enough space afterwards.especially when you take into account that the raise will happen during a very long time period and the flooded areas are really not that much of a deal.see for yourself>http://flood.firetree.net/in the worst case scenario theres still antarctica and where do you see a problem there i do not know
1. Both changes in existing society and establishing the new one are possible; neither can be realistically done without army, police, parliament, courts etc.
2. The main reason not to accept the refugees is they look and talk different and will require a lot of resources to resettle. The resources that will be needed for citizens - and the refugees aren't.
3. Antarctica with its sub-freezing mean temperatures and no soil is not so good for agriculture even if world's average goes up by 10 degrees. See?
well i agree that antarctica is a rediculous proposal.but i was merely asserting it to the possibility and your assumption, that the whole of vietnam and thailand would be flooded, which would require a tremendous rise in temp.in the previous link you can see that even a 50m rise would not cause such an armageddon.
I believe that the debate is not over!
Maybe we should actually have one. Al Gore vs. ?????
Ok. That one's too easy. Who would you suggest?
David Friedman Writes:
"Have you bothered to read the IPCC reports? Any idea of the magnitude of the changes in temperature and sea level that they suggest? Do you care whether particular claims make sense or are supported by evidence? Are you bothered if "your side" makes confident claims and then revises them when they don't come true?"
Nobody does this. Mankind is not composed of scientists, it is composed of primates. All primates take sides, all primates have hierarchies. You are asking this poor fellow to sublimate years and years of ingrained evolutionary programming that tells him to deal with any potential danger by FIGHTING or FLYING! The news says the world is going to end from global warming? FIGHT FIGHT FIGHT! BUY A PRIUS! Somebody tells me that my Prius isn't going to prevent global warming? He is interfering with my FIGHT! He will be indirectly responsible for the END OF THE WORLD AS WE KNOW IT! FIGHT FIGHT FIGHT!
Even PhDs do this all the time. I'm an AI and Machine Learning student at GA Tech, and I guess they've had a recent spat of doctoral students overfitting the training data, so they really hammer it into us: cross validate, cross validate, cross validate.
In cross validation, you take your training data, and randomly sample a subset of the training data--say, a statistically significant sample (>30 items), although in computer science, we would typically use a larger sample due to computing power--and use the remainder of the training data to validate your model.
Then, you rinse and repeat a statistically significant number of times (10 is usually enough, but statistical significance is technically >30x). If your trainer performs well, then you *might* have a good model.
On the other hand, the *very bad* *awful* *hugely embarrassing* (at least in computer science, I don't know about climate scientists) practice is to train on all of your training data. This is a huge mistake, because it causes overfitting. If you overfit your training data, typically your performance on new data is awful and outrageous.
Hence the "hockey stick" of naive climate models: they overfit the training data and (I would guess) observe the hockey stick some percentage of the time (the other times, they probably get a cooling hockey stick, but they don't mention those results...). If stock market algorithms, strategies, and machine learners did this, we'd get a market crash so fast it isn't even funny. It's amazing that when it really counts (when a mistake could cost you money) people don't eff around.
Anyhow, with people who don't know much about statistics holding the purse strings, there is a strong moral hazard for scientists to release these hockey stick predictions as valid results (even if they know they are invalid)...And there is the additional social hazard: a scientist who releases data supporting global warming is going to get a lot of attention and love.
Maybe even a Nobel prize (the loosest Nobel prize since Obama's).
Anyhow, we've got nothing to worry about.
Plus, when the dinosaurs were around, the Earth was 4 degrees warmer. The consequence was that we had an atmosphere with 30% oxygen or something. Think about the things we could do as a people if we had that much more energy! We should be actively trying to warm the globe...
The real concern is the macro-engineering task of moving the Earth's orbit away from the sun, because in a few million years the sun is going to be hotter. LoL.
On a final note, coming from physics and working as a research scientist, I have never met a scientist who didn't think that global warming was a crock of warm, brown soup. But I've never met a global warming article that didn't justify it's position by pointing to a nameless collection of "scientists" or "experts."
In physics class, on multiple occasions, we actually 'proved' that global warming couldn't happen. We did it by analyzing the wattage of the solar output, but the most memorable one for me was when we were studying thermodynamics.
You see, when the ocean warms up, it expands. The ocean's ability to absorb CO2 from the atmosphere is dependent on the surface area of the ocean (a lot) and the ocean's temperature (only a little).
The problem with global warming models is that they all depend on CO2 content in the atmosphere...yet any increase in temperature increases the surface area of the ocean so much that any excess CO2 is absorbed...
And of course, the excess CO2 will cause an algae bloom, so all the carbon will be captured into sugars and reintegrated into the biosphere. The left was right: it's all about the 'circle of life' :)
The hockey stick is a classic artifact of overfitting data to a computer model.
As an AI and Machine Learning student at GA Tech, this is basically all we talk about: you come up with a smart idea for a model, and you train that model with data. If you train the model with all of the training data you have, you will "overfit" the data and your model will draw outrageous conclusions given novel input.
But in a field like climate change, publishing the hockey stick will get you funding, so given choice between losing your job and getting lots of money, interviews, and getting laid, you are always going to publish the shoddy science.
The huge moral hazard is an embarrassment to the scientific community. Coming from physics as an undergrad, and after working for several years as a research scientist, I have never met a scientist or engineer who thought global warming and climate change wasn't a crock of something warm and brown...and yet "believers" in climate change point relentlessly to scientific consensus...
It is basically an unspoken fact in science that softer sciences--sciences that don't actually have to generate results--rampantly abuse statistics and pervert results.
Climate change science is a softer science. Get over yourselves, libs. Your "beliefs" are groundless and, in liberal language, "substantially irrelevant, dispositive to reality, and untethered to sound scientific practice, both specifically and generally:" we've already burned half of all the oil on the planet, and the most favorable climate estimations are that we have increased the temperature by a measly 0.7 degrees. Why would burning the second half be any different? Oh yeah, the hockey stick...but the hockey stick is the result of overfitting the data...so now what?
As a simple thought experiment, consider the dinosaurs. The dinosaurs lived on a planet that was 4 degrees warmer. If runaway warming of the Earth was possible--if the Earth was within the greenhouse distance of the sun--then it would never have cooled down. Further, when the Earth was a CO2 nightmare, with zero life and volcanoes everywhere, it cooled down. How is this consistent with a hockey stick?
In fact, the most remarkable thing is the climate models' dependence on CO2: in thermodynamics back in undergrad, we actually worked out that it was essentially impossible for CO2 to build up in our atmosphere if it also raises temperatures.
As temperatures go up, the ocean will expand. The increased surface area of the ocean will cause it to absorb more CO2...so as CO2 goes up, CO2 absorption goes up. Algae grows because of the CO2 increase, gobbles up the CO2, turns the carbon into sugar...no more CO2. This absorption would increase even more dramatically if the size of the ocean increases: so the more ice that melts, the more CO2 gets absorbed.
There is simply no logical reason to "believe" that a hockey stick is a reasonable prediction...and a perfectly logical reason to dismiss the hockey stick as sensationalism due to overfitting and moral hazard. Ignoring this is, IMHO, on par with the doomsday people quoting revelations in the bible: it is an act of faith.
Post a Comment