Climate Audit

bender

This approach is more productive than doing what Gavin did – computing an envelope for the family of models.

Although it does make sense to me to think of these models as samples from a population of possible models. It’s just that it doesn’t get you very far. With Douglass’ method here you are now forced to argue whether the models that fit the observations are superior, and whether this support is then sufficient to define “the fingerprint”. And that’s progress.

Now, who will be the first alarmist to step up and admit that the three models that fit best are the ones that happen to have the least credible convection parameterizations? Judith? Phil? Mr Crickets?

bender

#5
I just defended it, in #3.

Raven

I don’t understand why this approach was not used from the start.

If I am simulating a transitor I look for the model that has the best match to reality over the frequency range that I am interested in. I do not take every possible model and use the average output.

rk

This graph is certainly illuminating. It does seem like developing a mean and variance based on the obs then testing the models for goodness of fit is a logical approach, although with such a few sets of obs whatever bands you have won’t be very tight.

I’m sure its easy to increase the error bars of the 22 models so as to account for any obs that you want. So to do it the other way around seems a little silly. The error bars from RC obviously can account for almost any observations, therefore the falsification issue is real. I think someone has already said this…but I’m just now catching up.

beaker

Ron #9: This seems to me to be a result of the difference between inconsistency and bias in the pre-existing statistical sense of the terms, rather than those defined in the original paper. The ensemble is biased (the mean does not give a good picture of the observed trend), but it isn’t inconsistent (the observed trend is not outside the range of values considered plausible, if not perhaps highly probable, by the ensemble).

beaker

David Douglass said:

We noted that previous papers and IPCC reports had introduced the concept of averaging over models. There is no scientific justification for doing so. The models are not independent samples from a parent population because the models are, in fact, different from each other in many ways.

I doubt this was introduced by the IPCC, averaging over models is standard practice in Bayesian statistics, it is called “marginalisation”.

As for whether the models are samples from a parent population, consider this interpretation: Assume that we could construct a generalised GCM (G^2CM) that could implement any of the individual models in the existing ensemble by changing a set of simulation parameters. Now there will be some uncertainty in the correct setting of these parameters (which correspond to assumptions about the physics), the natural thing to do would be to suggest a statistical distribtion over these parameters establishing the plausibility of different settings. We could then construct an ensemble by drawing sets of parameters from this distribution and running them throught the G^2CM. In that way the ensemble is a sample from a parent distribution. The ensemble we have also could be considered as a sample from a parent distribution, although it has been created in a more ad-hoc manner and the distribution over plausible parameter sets (i.e. physical assumptions) is implicitly represented by the opinions held by the teams constructing the models. In principle, you could get the same result using the G^2CM approach, so it is reasonable.

beaker

Willis #14: Two additional things need to be taken into account:

Firstly the means for the runs for each model only really represent the systematic uncertainties involved (i.e. uncertainties in the underlying physics), the stochastic uncertainties also need to be taken into account, e.g. by taking the standard deviation over all model runs in order to reflect the true uncertainty of the ensemble. Certainly there is a problem to do with the fact that different numbers of runs were performed for each model. Taking the standard deviation over all runs is perhaps a conservative appoach, that is true, however taking the standard deviation over model means is an obviously biased approach as it ignores the stochastic uncertainty entirely. If you want to falsify a proposition, it is important to avoid unneccessary biases in favour of falsification, so we can be confident that the falsification itself is robust.

Secondly, as demonstrated by your plot, the estimates of the observed trend are not in particularly good agreement even with eachother, which shows there is uncertainty in the observed trend. To test for inconsistency properly, standard practice would be to show that the mean +-2sd error bars of the models don’t overlap with the mean +-2sd error bars of the data. Yes, I know that makes the inconsistency test easier to pass, but that is the correct way to perform the test for inconsistency.

Now if you want to test for bias, that is another matter entirely.

beaker

Willis #14: Two additional things need to be taken into account:

Firstly the means for the runs for each model only really represent the systematic uncertainties involved (i.e. uncertainties in the underlying physics), the stochastic uncertainties also need to be taken into account, e.g. by taking the standard deviation over all model runs in order to reflect the true uncertainty of the ensemble. Certainly there is a problem to do with the fact that different numbers of runs were performed for each model. Taking the standard deviation over all runs is perhaps a conservative appoach, that is true, however taking the standard deviation over model means is an obviously biased approach as it ignores the stochastic uncertainty entirely. If you want to falsify a proposition, it is important to avoid unneccessary biases in favour of falsification, so we can be confident that the falsification itself is robust.

Secondly, as demonstrated by your plot, the estimates of the observed trend are not in particularly good agreement even with eachother, which shows there is uncertainty in the observed trend. To test for inconsistency properly, standard practice would be to show that the mean +-2sd error bars of the models don’t overlap with the mean +-2sd error bars of the data. Yes, I know that makes the inconsistency test easier to pass, but that is the correct way to perform the test for inconsistency.

Now if you want to test for bias, that is another matter entirely.

N.B. This post was blocked as spam the first time, but I can’t see anything in the content that is objectionable.

Steve: The Spam KArma log stated: -10: Fake Javascript Payload.

JamesG

Beaker: You’ve just given us yet another reason to distrust Bayesian statistics. It doesn’t matter how much you say something is reasonable or plausible, or even if it has been done before – it doesn’t make it correct. Averaging across models is meaningless to a computer modeler. It doesn’t help drive our understanding of the model. Since Gavin is such a beast, he can only be using this standard deviation gotcha as an excuse to avoid seeing the obvious. The ensemble plot tells us all we need to know, using our brain. A modeler usually deals with whether the model is correct or not. Any test, statistical or otherwise which doesn’t help separate good from bad is useless. I’m glad we agree though that leaving statistics out is actually a step forward and more useful. I wish RC would realize that.

beaker

JamesG: Consider this, why doesn’t someone publish a paper showing the full uncertainties of the model, and say “have you seen the size of those error bars, do people realise just how uncertain the models are? Are models this uncertain really useful? Note also that the models have a significant bias, so the ensemble mean is not a good fit for the observations”. That would be a really good way of pointing the important defficiencies of the models, and it would have the full backing of both frequentist and especially Bayesian statistics.

Note the revised version of Willis’ test that I give in #19 is fully frequentist. A Bayesian would probably prefer to look at Bayes factors for competing hypotheses rather than a conventional significance test on a single null hypothesis.

beaker

MarkW says:

If I were to take a Semi, trailer fully loaded, a Semi, no trailer, a NASCAR stockcar, a nitro burning dragster, a Chevette, and a Sherman tank, and average their 0-60 times, what would that tell me about the capabilities of the internal combustion engine?

That we can’t give confident predictions about 0-60 times unless we have a good understanding of the vehicle, i.e. there are large error bars on the 0-60 times. Equally, if we have imperfect knowledge of the physics of climate change, we will end up with models that make very uncertain predictions, with very wide error bars – and that is just what we have! It really is puzzling to me why it is so hard to see that it is the uncertainty of the models that is the greater practical problem that should be recieving a greater focuss.

James Lane

I recall looking at the Douglass paper and the RC response some time ago and chuckled at how Gavin suddenly discovered bucketloads of uncertainty in the GCMs to refute Douglass et al.

But I’m rocked by what I’ve read in this and the trop trop thread. I think JamesG puts it well:

Since Gavin is such a beast, he can only be using this standard deviation gotcha as an excuse to avoid seeing the obvious. The ensemble plot tells us all we need to know, using our brain. A modeler usually deals with whether the model is correct or not. Any test, statistical or otherwise which doesn’t help separate good from bad is useless.

Even beaker seems to have conceded that the models are hopeless for the tropical troposphere, although maintaining that they are not statistically falsified.

beaker

James Lane says:

Even beaker seems to have conceded that the models are hopeless for the tropical troposphere, although maintaining that they are not statistically falsified.

I’m sorry, but this comment is an illustration of the problem with blogs on climate change, which is that everyone is automatically assumed to have a position on one side of the debate or another. I don’t, however I do think it is important that papers potentially containing errors are audited in an impartial and impersonal manner, without any commitment to the outcome of the audit, regardless of which side of the debate the belong to. I have made no such “concession” as I have not said that the ensmble means give good predictions of the tropical trophospheric trends (and neither for instance does Gavin@RC if you read his article in an unbiased manner)!

The models have NOT been falsified, they have been shown to be significantly biased, I have explained the difference (and at Willis’ request demonstrated that these terms have pre-existing meanings in statistics). The reason why Douglass et al. needs auditing is because some, such as yourself, believe that inconsistency has been established rather than bias, it hasn’t.

beaker

Judith: For those interested in interpreting the IPCC scenarios for practical applications, Bayesian statistics already provides a sound means of intepretation. Just run each member of the ensemble through the impacts model separately and collect the results. The distribtuion of the impact variable that you get this way is the logical consequence of the uncertainty in the IPCC scenarios. Any impact that is within that distribution is plausible according to the scenarios, the mean of the distribution gives you the expected (in statistical sense) impact. I have investigated this approach myself and it does work, the only problem is you have a hard time explaining it to people who don’t trust statistical methods.

If the scenarios are uncertain, you get uncertain predictions of the impacts. I would find that reassuring as a statistician in any means of interpretation.

If you just run the ensemble mean through the impacts model, the error bars then only reflect the uncertainty imposed by the impacts model whilst ignoring the uncertainty of the GCM output. As a statistician, the narrow error bars would worry me.

Paul Linsay

Looks at Willis’ plot in #16 here and #144 on Tropical Troposhpere thread then reads this in #29

The models have NOT been falsified, they have been shown to be significantly biased…

Reminds me of Grouch Marx’ question “Who are you going to believe, me or your lying eyes?”

Lance

After originally watching beaker with a jaundiced eye I have concluded that he is just insisting that a rigorous definition of “falsified” be applied to the issue. He has submitted that the ensemble of models has not been “falsified” in a Bayesian sense.

But speaking in Bayesian terms one must ask what is the “degree of belief” that can be attributed to the predictive power and thus usefulness of these models based on the statistical state of this analysis.

Being a lowly physicist with little statistical acumen I haven’t the ability or proclivity to make such a determination.

Perhaps beaker could render an analysis that could estimate the Bayesian probability of these models having useful predictive skill.

beaker

Steve: Absolutely, the standard deviation of the model outputs is not an estimate of the stochastic uncertainty of the Earths response, it is a measure of the uncertainty of the prediction, taking into account all of the uncertainties, including the variability of the thing we are trying to predict as well as the uncertainties involved in constructing and estimating the model. However, if you want to invalidate the model, you need to show that the models can’t give rise to the observed data, i.e. the models don’t think the observed data are plausible. It is the standard deviation error bars that define the range of values that the ensemble considers to be plausible (taking all uncertainties into consideration). The SE error bars only show the uncertainty on the expected value of the trend, not the uncertainty of the modelled trend itself.

I’m not sure it is possible at this stage to estimate the true stochastic uncertainty of the Earths response using the models. It isn’t my area of expertise, but if they can’t reproduce a trend that the theory apparently suggests they should show clearly, it may be an indication that the systematic uncertainty is too dominant. But that is just my intuition.

The ensemble as we have it now is already the result of a-priori judgements on the part of the modellers. The ensemble basically reflects the range of ways in which the climate works that the modelling community consider plausible. If a number of groups independently come up with similar models, it is an indication that a particular set of assumptions is more plausible than others. If the ensemble is broad, this just reflects the uncertainty within the modelling community on the basic physics of the climate. The last thing we should do is ignore this by concentrating on the mean all the time.

beaker

Andrew: Looking at Willis’s figure in #16, I’d say they disagreed substantially rather than only slightly, if I understand it correctly, compare e.g. RAOBCORE and IGRA trends at 200 (hPa).

kim

Well, I meant doubling of carbon dioxide in the atmosphere.
==================================

bender

beaker’s recent contributions #13, #17, #19, #23 etc. are not sophomoric. I agree with everything said here.

bender

The models have NOT been falsified, they have been shown to be significantly biased.

Puh-leeze. To remove the bias requires that you adjust the model implies you think the old model was wrong implies that the hypothesis was rejected implies the model was falsified.

Word gamers.

bender

Raven’s #2, folks. beaker, do you know the answer?

Sinan Unur

At first, I was troubled by the fact that each GCM was run a different number of times. I was under the impression that that was a choice by Douglass et al. More careful examination of the references led me to https://esg.llnl.gov:8443/index.jsp (which seems to have an incorrect SSL setup).

If I understand correctly, those were the runs used for the IPCC reports. Therefore, it does not make sense to try to choose among them on the basis of some ad hoc criterion but to use the results which were used to make policy recommendations.

I need to decide whether to get an account with the intention of writing a paper. Of course, before I can decide whether to do that, I need to see how much work it is to deal with the datasets. For that, I need an account. Seems a little circular 🙂

Any way, the point of this post was to state that I no longer find the multiple and different number of runs per model that troubling given that it no longer looks like the decision was up to Douglass et al.

Of course, I am a little slow, so this might have been obvious to everyone else.

— Sinan

Pat Keating

46 bender

SOME OF the model results

If you wish, although I would point out that none of the models is close to the observational data at the most-important uppermost (100 hPa) part of the troposphere (where, BTW, the observations are quite close to each other).

beaker

Pat Keating says:

However, my main point is that while statistics can be used to illuminate, it can also be used to obfuscate, and the clear difference between model and observational data in Fig. 1 is being lost in the discussion. This has been used to claim that the paper is “fatally flawed”. Flawed, perhaps, but fatally so, surely not.

One way of obfuscating with stats is to use phrases with pre-existing statistical interpretations (such as “inconsistent”) to mean something else, when there is already an existing statistical term with exactly the desired meaning (“biased”). The problem I have with the paper is that it makes a very big claim (inconsistency) but performs a test that can demonstrate only bias, a lesser claim (I have lost count of the number of times I have had to type that ;o). If they want to show the ensemble mean is a poor predictor of the observational data, that is a meaningful test of whether the ensemble is useful (although not the only one), and they have made their case. However bias and inconsistency are not the same thing, and they claim inconsistency.

If Douglass’ test falsifies anything, it is the use of the ensemble mean to make predictions without considering the uncertainty. But then again, any decent statistician would tell you that is inherently the wrong thing to do anyway on basic principles.

bender

#53
beaker, get over it. Agreed: the Douglass paper is not perfect. The point is: perfection is not required. They have made a very important observation and the alarmists are trying to hand-wave it away. You clearly agree that the observation is important. You can use words to distance yourself from the skeptics if you like, but the fact is you are as skeptical as we are about these models. Otherwise you wouldn’t be parsing the paper and this discussion as carefully as you are.

Eggplant fan

Beaker #53:

I’ve checked a number of the the statistical terminology glassories listed at http://www.glossarist.com/glossaries/science/physical-sciences/statistics.asp?Page=2 and have not found an existing definition of “inconsistent”. Douglass et al.‘s use was informal, but was it really improper?

beaker

Steve: Re: “Beaker’s original endorsement of Gavin’s position is valid, but that’s entirely different.” I think my endorsement will depend on how you read Gavin’s article. My interpretation (as a fairly neutral observer) was that he considers “inconsistent” the same way that I (and other statisticians) do, as meaning that the ensemble can’t explain the observational data, i.e. the ensemble considers the data to be outliers, or implausible. In that case, the +-2SD test is the right test. My reading of the article was that he also agrees that the model fit is not perfect (I would go further and say it is fairly poor unless you look at RAOBCORE 1.4). All he argues is that there is no clear inconsistency between the models and the data in the usual statistical sense, and he has demonstrated that. I am reasonably confident, having read a number of his articles, that if asked, he would readily agree that the ensemble demonstrates a significant bias.

If you consider Gavin’s position to be that the models recieve any significant corroboration from these observations, then I would not endorse that! There is plausibility, but little support.

bender

#50 For pure scientists, yes, statistics are always preferred over heuristics. But to the extent that global policy is run by business interests and bureaucracies who have no time for “paralysis by analysis”, the world is indeed being run on heuristics. Should IPCC be held to a higher standard? Some would argue that. Those in favor of a “precautionary principle” would argue otherwise. Funny how it’s always the other guy that needs to show more “common sense”.

Your other points have been made many times and are tiresome.

Christopher

Re: 62

beaker, I understand that. I had, perhaps mistakenly, thought there might be a different (more Bayesian) approach you would adopt.

beaker

Eggplant fan says:

I’ve checked a number of the the statistical terminology glassories listed at http://www.glossarist.com/glossaries/science/physical-sciences/statistics.asp?Page=2 and have not found an existing definition of “inconsistent”. Douglass et al.’s use was informal, but was it really improper?

You won’t find inconsistent defined that was as an individual term, as it has another deeper meaning as a jargon term. However it does have a clear statistical implication , I posted an example of this on the other thread and gave full details of the paper. You can find plenty more by performing a Google Scholar search for “inconsistent with the data”. Incidentally, yesterday I discussed this with another statistician in my department, over a cup of tea, this time a proper frequentist one of considerable experience, and he said that he would naturally interpret “inconsistent with the data” in exactly the same way that I did (which was reassuring ;o) (and smiled when I explained how it has been used in the paper). Having discussed this issue now on CA, I can see why the term was being used, but it is improper in a statistical sense, as there is already a more accurate and specific term (bias).

beaker

bender says:

a decent statistician would tell you not to use the ensemble mean without considering the uncertainty (+-2sd error bars) anyway

1. So how would YOU do that?

I have already explained that a couple of times in the last day or so and at least once on this thread. See e.g. #31.

The models give an impression of the logical consequences of the detailed assumptions regarding climate physics on which they are based, nothing more. If we do not understand the climate physics really well, the predictions are likely to be unreliable. However, their predictions can give an indication of the range of outcomes that can be considered plausible based on what we do know. Which is why we need to look at the whole ensemble, not just the mean, if we are really listening to what the GCMs are telling us.

If you understand that, you will see why Gavin is right in the test of inconsistency, and why establishing bias is not neccessarily the only test of whether the GCMs are useful.

Patrick M.

(Beaker):

I’m a layman trying to understand your descriptions. Would I be correct in making the following analogy?

If we were talking about shooting a rifle would:

1 – uncertainty = grouping
2 – bias = distance of the center of the group from the bullseye

If this is accurate, how would this analogy work regarding GCM’s?

Thanks

beaker

Patrick M: Spot on. As an analogy, the models all predict different values for the trend, the spread of these predictions is just like the grouping and the bias is very much like the distance from the center of mass of the group to the bullseye (except that there are uncertainties in the data as well, so we don’t know exactly where the bullseye is, although we have a pretty good idea).

The idea of inconsistency would be like saying that the marksman is so bad that the bulseye isn’t even anywhere within the grouping.

The idea of significant bias is basically like saying that the bullseye clearly isn’t in the middle of the grouping, so there is good evidence that the marksman needs to adjust his sights (or the climate modeller his GCM, as noted by Gavin on RC).

MrPete

I found this helpful, and enjoyable, not least:

3) A frequentist is a person whose long-run ambition is to be wrong 5% of the time.
4) A Bayesian is one who, vaguely expecting a horse, and catching a glimpse of a donkey, strongly believes he has seen a mule.

🙂

PI

Some people seem to be confused about the Bayesian approach to this problem.

beaker initially said that averaging models is a standard Bayesian practice, and JamesG complained that was therefore “another” reason — I wonder what the rest are? — to distrust Bayesian statistics.

What beaker described is technically Bayesian, but it uses rather unrealistic prior assumptions: it assumes that you believe that all models are, a priori, equally probable to be true.

Nobody actually believes this. But if you did believe it, then averaging the models would be perfectly valid, so this doesn’t speak against Bayesian methods: it’s a correct inference, given that assumption. The problem is that we have experimental and theoretical reasons to believe that some models are more probable than others, not that Bayesian inference doesn’t work.

What Bayesians actually do in Bayesian model averaging (BMA), a la Raftery, is to give each model a probabilistic weight and compute a weighted average of the models. The weight is determined by how well each model fits the observations, as quantified by a likelihood function. (There is much room to argue about what likelihood to use, but frequentists have that problem too.) Models which fit the observations have a higher posterior probability of being true, and get a greater weight in the average.

This is, in essence, a fancier version of what Douglass is describing in frequentist terms. Models which don’t fit the data are essentially discarded (given a low posterior probability). If you choose to average them, those models contribute little to the ensemble average (as long as there are models in the ensemble which fit the data better).

Not all Bayesians agree with the BMA approach to model combination; there are different camps who like different approaches. BMA assumes that there is such a thing as a true model, and that one of the considered models is the true model (but we’re not sure which). Arguably, this is not the case. BMA proponents have various responses to that objection. One is that it’s not rigorously correct but it is a good approximation as long as the considered models span, in some sense, the full space of reasonable models. Another is more pragmatic, in that the model average is often demonstrably a better predictor (according to, e.g., cross validation) than is any individual model, even when all the considered models have obvious structural errors. There may be other, better responses; I am not an expert in BMA.

beaker

PI: The problem with re-weighting like this is that the historical data has already been used (even if not directly) to tune the model parameters, so you would be using the data twice, which raises problems. Assigning equal weight to each model at least expresses an equal confidence in the ability of each modelling group. Not being an expert in the field I am not qualified to suggest a more realistic prior than that, but I can advise on how to procede if a better prior were available. However my intuition would suggests an equal weighting would guard against making extreme predictions in either direction, which would be no bad thing as it is generally better to express uncertainty than undue confidence.

Christopher

Re: 75
Thank you, that was very useful (in effect what I was hoping to glean from beaker). Again, my Bayesian ability is limited but the reason I’m curious here is I see a link between ensembles and ergodicity. This is a real question in my opinion. I have also found an example where this procedure was applied in an relevant domain and where some of the issues here are discussed with more rigor that Douglass: “Using Bayesian Model Averaging to Calibrate Forecast Ensembles.”

beaker

Pat Keating says:

59 ..he considers “inconsistent” the same way that I (and other statisticians) do, as meaning that the ensemble can’t explain the observational data, i.e. the ensemble considers the data to be outliers, or implausible.

This is trying to turn science upside down. The models are the “outliers” or “implausible”, not the data. It is the data which is pre-eminent, and a useful model must explain the data, or it fails

If you actually read my posts, you will find that I have repeatedly said that consistency does not imply usefullness (it is a neccessary, but not a sufficient condition). If you read Gavin’s RC article, you will find that he also says that the lack of inconsistency does not mean there is a good fit. Language like “trying to turn science upside down” is unhelpful, especially when it is Douglass et al that have redefined (I expect inadvertently) the meanings of existing statistical terms, not gavin or I.

Pat Keating

81 beaker

I objected to the language you used (the ensemble considers the data to be….implausible.), which strongly implied that model results had at least as high a standing as observational data.

I suspected that you didn’t believe that, but such biases in language, even though inadvertant, are important in science, which relies on careful statements for good communication. (BTW, an ensemble cannot consider anything).

Steve McIntyre

RAOBCORE versions are another controversy regarding this paper. RAOBCORE http://www.univie.ac.at/theoret-met/research/raobcore/ is an acronym for “RAdiosonde OBservation COrrection using REanalyses (RAOBCORE)”. So it’s a radiosonde version of the adjustment process and not independent measurements.

They use HadCRUT3 surface temperatures in their calculations and thus are not “independent” of HadCRUT3 to the extent that that matters:

For the calculation of the MT and LT layer mean temperatures, not only radiosonde data but also HadCRUT3 surface temperature data have been used.

The tropical troposphere appears to be a source of concern for them:

Cautionary note: The tropical mean trends 1958-1978 show warming at low levels but cooling at upper tropospheric levels, which seems unrealistic. This feature is related to a strong warming anomaly over the Eastern US which is spread to the Caribbean by the ERA-40 bg. Consequently the Carribean stations may be overcorrected at low levels during this period. Since most tropical stations are in the Caribbean during these early days, this problem strongly affects also the global tropical means. This issue, which affects all RAOBCORE versions, will be fixed in the next version.

Their web-visualization diagrams show that one of the outcomes of their adjustment process is in fact to increase the trend in the tropical troposphere.

In a quick look at their adjustment procedures, they look a lot like what USHCN plans to introduce – heavy reliance on changepoint analysis developed by climate scientists. An important issue to be sure, but I’m pretty sure that the USHCN changepoint analysis is easily contaminated and this is something that would need to be examined for RAOBCORE.

Jedwards

Re #71, Ok that was very clear and understandable (being trained in classical marksmanship myself).

However, your statement below is actually what we have here. The grouping of the ensembles clearly misses the bullseye (demonstrating bias or “windage” in marksmanship terms), but appears to at least be on the paper (and therefore NOT inconsistent).

The idea of inconsistency would be like saying that the marksman is so bad that the bulseye isn’t even anywhere within the grouping.

As anyone who has fired a gun will tell you though, its pretty obvious that these folks need to “adjust their sights” more than just a touch.

beaker

Pat Keating says:

I objected to the language you used (the ensemble considers the data to be….implausible.), which strongly implied that model results had at least as high a standing as observational data.

I suspected that you didn’t believe that, but such biases in language, even though inadvertant, are important in science, which relies on careful statements for good communication. (BTW, an ensemble cannot consider anything).

Language biases are indeed important in science, which is why I have been using the correct statistical terminology, unlike Douglass et al., which is then repeatedly misunderstood! Plausibility is about possibility, not probability. If I had said the ensemble considers the data to be probable that would suggest that the data provided a degree of support to the models. If I said the ensemble considers the data to be plausible, that means only that the data do not contradict the model, but that does not imply any substantial degree of support. I chose my words there carefully. I find the best thing to do in a discussion/argument is to give my opponent the benefit of the doubt and try my best to see his point of view. A good start is to look for the most moderate interpretation of what they have said, and use that, rather than an extreme one (note you say you expected I didn’t mean that, but your response clearly implied that I did).

BTW, from a Bayesian perspective, the output of a model (or ensemble of models) expresses the distribution of belief over a set of propositions or outcomes, so it is not that unreasonable to say that the models consider something plausible. Of course I could have written it more formally, but given that the scientifically correct terms that I have used so far have been repeatedly misunderstood, even though I have explained fully what they mean several times, I’m not sure that would be a good idea!

beaker

PatrickM #88 That sounds like a reasonable summary (although the chart in #16 doesn’t include the stochastic uncertainties in the models or the uncertainty in the data, and I think all the plots are normalised to the surface trend whichif fixed would all make inconsistency harder to prove).

Inconsistency is defined at a level of confidence, so at 2SD you can be about 95% sure that there is an inconsistency, 3SD you can be 99% sure. The more standatd deviations you use, the more confident you can be of the inconsitency, but at 3SD you are really already at the point of diminishing returns for most purposes.

beaker

PatrickM. Cheers! At last I can go home for the weekend and enjoy some cricket content in the knowledge that somebody understood exactly what I was saying! ;o)

Have a good weekend everybody.

P.S. I am generally for GCMs as AFAICS there isn’t a better approach, but they do appear to need some work!

Steve: I played some cricket when I was at university in England (hadn’t played in Canada.) To this day, I remember the first over that I ever batted. We had a weak college team and I played mostly to fill out the ranks, but I could field and throw well. Because I hadn’t played before, they placed my last in the order and I never got up. One slow day when we were getting killed, I asked to be moved up the order – the other low batters weren’t any good either so I couldnt do any worse. The other team was using a pretty weak bowler as well. I swung like a baseball player and got 16 runs on 4 home runs or whatever they’re called. So the other team immediately changed bowlers and the next bowler gave some complicated spinning thingee which turned me inside out and that was it. The cricket fields in England were pretty civilized places on a spring day.

Pat Keating

85 Steve M

For that reason, I prefer the Had-AT data. I believe this is from radiosondes using conventionally-calibrated thermometry and minimal “adjustment”, though I’m not sure of that.

PI

Pat,

It’s not accurate to say that the Bayesian perspective means that the models don’t have to justify themselves against the data. The whole point of Bayesian inference is to infer the relative probabilities of hypotheses (here, models or model assumptions) as implied by the data.

In my opinion:

The phrase “the model implies that the data are unlikely” should technically be interpreted as a statement about the likelihood function, p(data|model). (It would be less ambiguous to say, “the model implies the data are unlikely, assuming the model is correct”.) Both frequentists and Bayesians use likelihood functions.

The phrase “the data imply the model is improbable” should technically be interpreted as a statement about the posterior probability, p(model|data). This is a fully Bayesian statement; frequentists don’t speak of the probability of hypotheses, only on the probability of data conditioned on hypotheses.

Patrick M.

re #96 (beaker):

LOL! You think #92 meant you could go?? Oh no, that just means we can consider you a resource, see #96…

Patrick M.

Correction

re #93 (beaker)

steven mosher

Beaker or Dr. Curry, as I noted in 86, when models with natural forcing only dont
match observations ( attribution studies) what type of statistical tests are used
to determine that the models ( with natural forcing only) dont match the observations?

Is this test done formally? or just visually.

Sam Urbinto

Now we see what happens when a bunch of people who basically agree with each other start discussing the details. 😀

I like computers.
Oh, which OS.
Windows. <—&rt; Linux.
Both scream at once, you suck!
Ladies and gentlemen, start your engines.

BTW The BB (gun) is at http://www.climateaudit.org/phpBB3/

NTaylor

Steve McIntyre (#96) stated, “wonder if a re-phrasing of the point in Douglass mightn’t meet Beaker’s standards.”

As a nonstatistician (clinician), why should there be a concern about meeting Beaker’s ‘standards’? Is it just an interest in another statistical method (theory/philosophy) and that this interests you from a technical standpoint?
In other words, for me, in reading most every post in the two applicable threads (began to lose interest in beaker’s certitude/certainty near the end), it appears that the key for his absolute pronouncements on the Douglass paper are entirely based upon a Bayesian approach.

My question relates to the validity/reliability (in laymen’s usage) of the Bayesian method itself. It is epistemological, I guess. Independent of appeal to authority, or that it is in ?widespread usage–is there any evidence or data (other than it’s own self-generated conclusions) that show it should even be given weight in this (or any other) discussion? Are there error bars, SD or SEM for the statistical method itself so that one can assess independently whether even looking at it from a Bayesian method is worth spending all energy on–except as an intellectual exercise?

In case there’s a question of motivation in this post, I’m certain (w/in 95% confidence intervals) that I’m asking a serious question (of course, except for this one!).

[My experience in my (medical) field has been that many treatments and/or approaches have been widely adopted before ever showing they actually do what they were “intended” to do; that many of these passed ‘statistical analysis’; and that relief from the propagation of the errors after being widely accepted as ‘true’ takes a lot of energy. That time/energy should have been better spent at the beginning being skeptical.

Also, as a side note; just based on the posted charts/graphs themselves, were this a meta-analysis of a medical question–as one who is responsible to my patients to first do no harm, I would never accept the conclusion that the models were useful. I wouldn’t care about the debate as to whether they were ‘falsified’ or not–they would be so for any implementation into my practice.]

NT

Pat Keating

97 PI

It’s not accurate to say that the Bayesian perspective means that the models don’t have to justify themselves against the data.

No, I realize that. However, the language Beaker used (based on “the Baysian perspective”) stated that the observational data was “implausible”, a truly topsy-turvy viewpoint. That is what I objected to, because the GCMs are already given unearned credibility IMO in many quarters, and this was taking it too far, when one reads it as stated.

Cliff Huston

Beaker,

The disconnect in this dialog is one of context. In terms of the proper statistic to use for a test of inconsistency of the model ensemble, that context was set by the IPCC report (as shown by Steve M in #52 above). For all practical purposes, the IPCC has put forth one model, the output of which is defined as the average of the ensemble. In that context the only choice is to use standard error for the model error bars.

You don’t agree with the IPCC defined model context, and you use that disagreement (in effect) to declare Douglass et al (who simply used the IPCC defined model context) ‘fatally flawed’. You may well be correct in this assessment, but you should then be as forth right in declaring the IPCC report ‘fatally flawed’. I doubt very much that Gavin (or the IPPC cast of thousands) would agree with that assessment of the IPCC report, but it would be logically consistent.

Ah you say, but the IPCC was not testing for consistency but rather they were merely discussing consistency. Douglass et al, on the other hand, were testing for consistency and had a duty to change the context (hence the meaning) of the model data and use standard deviation for the test. It matters not that they were testing the IPCC theory that the model output (defined as the average of the ensemble) would skillfully track observed temperatures in the troposphere.

But how can that test be valid, when it is out of context?

Cliff

Patrick M.

re #108(Michael Smith):

It seems you could do one of those robustness deals where you recalculate the +-sd2 with various models removed and show that passing the inconsistency test depends on one or two models being present, (like what they did with the bristle cones).

Would that be a statistically valid approach?

Lance

Jon,

Your complete dismissal of Douglas et. al. seems a bit overwrought. The issues raised here by beaker and others seem to be mostly semantic. The predictive skills of the models used by the IPCC while not slain by this study are certainly on life-support.

Willis Eschenbach

beaker, you say:

To test for inconsistency properly, standard practice would be to show that the mean +-2sd error bars of the models don’t overlap with the mean +-2sd error bars of the data. Yes, I know that makes the inconsistency test easier to pass, but that is the correct way to perform the test for inconsistency.

The fact that #16 does not show the +-2sd for the data means that we haven’t shown a proper test for “inconsistency”.

My friend, you are moving too fast. Think about it for a minute.

If all of the observations are outside the ±2SD error bars for the models, the ±2sd error bars for the observations will also be outside of those error bars. Look at it again, and this time consider where the ± 2SD error bars for the data will go.

Therefore, I have shown the test that you required. The observations (and thus their error bars) are everywhere outside of the models ±2SD error bars. By your and Gavins definition, we can say that the models are inconsistent with the data.

As near as I can tell by re-reading both threads, that was your only point. Now that your point been proven wrong by the results of the inconsistency test, can we move on?

w.

Andrew

116 (Bob B):

Jon, while Douglas was not successful in showing there is no AGW fingerprint, I would dare say any reading his paper came away doubting the models are capable of “predicting” the Earths temperatures with any certainty. They are useless especially where political and policy decisions need to be made. Douglas has more then cast a doubt on the physics behind the models, since there is no clear AGW signal.

Whuh?

Bob B

Let’s frame the discussion the other way. What is the confidence using the data from Douglas that the models show skill in predicting the AGW footprint?

Dave Andrews

Jon,

You must surely know that this isn’t just about the Douglass paper. The models have considerable flaws, there are so many things they can’t parameterize in any meaningful way. They might be the best that can be achieved at the moment but they are far from adequately recreating the reality of the earth’s climate.

This is probably accepted in the climatology world but it is not the way things have been presented to the general public, nor is it the impression IPCC gives to policy makers. Political decisions which impact millions of peoples lives are thus being made on the basis of inadequate models.

bender

Don’t let beaker off the hook yet.
1. He has not explained why he thinks there is a bias in some of the GCMs.
2. He has not expained if tropical tropospheric warming is or is not part of the AGW fingerprint.
3. He hasn’t explained why GMT has flattened since 1998.
He’s half-answering all the easy questions, dodging the important ones.

bender

A simple set of yes/no questions: was the Douglass et al. focus on the tropical troposphere justified? Is it now? And now that it has been admitted that vertical convection parameterizations are shaky (where does it say that in AR4?), would a focus on tropical troposphere be difficult to justify?

This is NOT about Douglass’ paper. NOT AT ALL. It is about the climate community – and RC’s – reaction to the paper’s revelations. Gavin’s response may have been, technically, error-free, dr beaker; however it was a dodgy kind of truth, NOT “the whole truth and nothing but the truth”. The whole truth would address the questions posed in #125.

bender

Jon, what is the “fingerprint”? Where do we look for it? Thank you in advance for not dodging this all-important question.

Jedwards

Re #117 Willis, I think we need one more chart, this time with MEAN_Obs +/-2SD included. If there is no overlap then I “think” the inconsistency claim is valid according to what I’ve been reading in this thread, according to beaker.

bender

#117 Willis

If all of the observations are outside the ±2SD error bars for the models, the ±2sd error bars for the observations will also be outside of those error bars. Look at it again, and this time consider where the ± 2SD error bars for the data will go.

Disagree, Willis. The upper bound on the data (+2sd) will be higher than the lower bound for the model (-2sd). i.e. They will overlap to some degree.

Bob B

You can stretch the statistical arguments to prove global warming is consistent with global cooling:

http://sciencepolicy.colorado.edu/prometheus/archives/climate_change/001413global_cooling_consi.html

twits

bender

#135
Cooling is evidence of “internal climate variability”. Warming is evidence of long-term GHG trend.

kuhnkat

It would appear from Beaker’s point of view that the main point of contention is the use of the word INCONSISTENT.

In his world inconsistent means that the models COULD NOT INCLUDE THE OBSERVATIONAL DATA AS AN OUTPUT, and the test does not show that.

In his terminology BIAS is what has been shown.

Although Douglass did give a definition in the paper for what he was showing, Beaker apparently feels this is insufficient to cover for the use of a term that has a specific meaning among Statisticians!! That Douglass was using statistics to make his point gives this some validity.

The quick fix would be for Douglas to amend his paper changing the terminology.

The harder fix is to require RC and/or IPCC to show the usefulness of using a suite in general, and this suite in particular, for this purpose in the first place!!!

Michael Smith

Cliff, thanks for clarifying that.

Bob B

Bender says:
May 2nd, 2008 at 4:37 pm

#135
Cooling is evidence of “internal climate variability”. Warming is evidence of long-term GHG trend.

And whee is your objective proof???

Bob B

snip – no ruminations on policy please.

Geoff Sherrington

Re # 49 Sinan Unar and similar

Any way, the point of this post was to state that I no longer find the multiple and different number of runs per model that troubling given that it no longer looks like the decision was up to Douglass et al.

We have seen statistical theory dissected in some detail above. But I repeat an earlier point. The figures being examined were almost certainly cherry picked by the modellers from far larger numbers of simulations that were never published or used in IPCC reports.

If you are going to spend so much effort on statistical dissection, why do you not contact the modellers and ask them for ALL of the results of ALL of their models, not just the ones chosen by them (by probably subjective/biased criteria) for submission to papers that ended up in IPCC compliations.

If you include ALL the simulations, including the rejected ones, I suspect that arguments about Bayesian versus classical, SD versus SE, distributions, falsification and so on would disappear in one huge bowl of well-mixed spaghetti that has a foul taste.

Max

Just a layman question, since the modern observed data is being argued as the flawed part of the comparison, how about modeling backwards, if you program a climate model, with data from the past from all the temperature proxies done, does it vindicate all the past temperature proxies by matching them all together along with all known forcings at the time? Do the predict the proper temp changes when major natural disasters occcured and put known elements in known amounts into the air? Or is that asking to much of the models?

Joe Solters

Bender #125. Bender is correct that beaker has been dancing on the head of a pin with his ‘inconsistancy’ schtik. And contributes nothing else. Likewise a lot of commentary, perhaps of necessity,by delving into statistical trivia, fails to recognize the extremely weak posture of current global climate science. We may be heading into a new 10 year or longer cooling period, but the IPCC climate models are useless for analysing this trend, or as this blog shows, any other observed trends or claimed climate fingerprints.

snip – please don’t vent about climate politics.

bender

We may be heading into a new 10 year or longer cooling period, but the IPCC climate models are useless for analysing this trend

They have done themselves and everyone a huge disservice in under-estimating and/or under-representing the stochastic “internal climate variability”. For example, I suspect lucia’s “refutation” is flawed – not through any fault of hers, but because you need an ill-behaved non-ergodic GCM to represent the true variaiblity in Earth’s cliamte.

And, yes, I am aware this is gross speculation.

For me, the question is just how poorly these models perform in representing the internal stochastics. The ocean scares me.

Ron Cram

bender,
re: 153

Would you say your view of the ocean has changed at all in recent months?

bender

#155 No. Why?

bender

#156

one could apply the beaker method (he’s sharp and thought provoking) to the attribution question

dr beaker, to the blackboard, please.

bender

#160
1. I never said, suggested, or implied this. Maybe you misinterpreted something I said?
2. PDO more powerful than ENSO? First, this is not a matter of opinion. Second, I think it can be disproven fairly easily based on physics.

Joe Crawford

Re #37 beaker: “If a number of groups independently come up with similar models, it is an indication that a particular set of assumptions is more plausible than others”

If we are going to be precise, this only holds true if there is no bias introduced by the scientists that generated these assumptions. Using your own definitions and methods, when taken as a group the 22 models are biased. The 64 dollar question is “Is this bias due to good science, or preconceived notions / group think of the scientists?” Since that question is imposable to answer (i.e., the models were not developed in isolation), the IPCC is wrong and each model and its results must be treated separately.

Joe

Jon

@138

This was not too long after you wrote this in the same comment

Again, that was in reference to a specific kind of post that finds a home here and at other blogs that let’s just say isn’t particularly credible. I didn’t call any poster or blog a “denier” and again, it’s pretty transparent to see attempts to imply such as a way to avoid my comments on this topic.

When you and others insist on treating Singer and Christy’s work in the context of the body of their work

Shockingly, I and others have a huge problem with the way Singer is choosing to comport himself. I would think, of all places, readers of this site would have zero patience for politicizing and grossly overstating the results of flawed science in order to advance a particular take on policy.

I have yet to see one person denounce Singer’s obviously unsupportable claims regarding this paper. I can only imagine what this thread would be like if the author was Mann. If Dr. Douglass returns to this thread, I would love to hear his reasoning for allowing his coauthors to purport this paper as being something that it clearly is not.

@142

Jon, must of us know what Beaker has contributed here and little of it has to do with his “auditing” picking out a term in the paper that the authors obviously were using in a general sense when the context is revealed.

As I said earlier:

1. The Douglass et al paper is fair game to be audited just like any other. As such it should be judged on its success or failure to achieve what it claims to.
2. It claimed to show inconsistency when it did not. Whether or not this was an error in understanding the language used or a genuine but mistaken belief that it did, the result is a failure to properly demonstrate the claim.

That the failure of this paper to demonstrate its claim is being so willfully excused by some- not all, and to the rest of you I tip my hat- is telling. Extremely so given the way in which it was presented by its own authors.

bender

And, Jon, Raven has a question for you in #2.
[Prediction: dodge.]

Craig Loehle

It seems to me there is some serious cart-before-horse issues here. When testing a model, we want the model to fall in the confidence intervals of the data. We do not accept a model is valid when it has wide ci and these barely touch the data. In the latter case, we can say that 95% of the time the model results are way outside the data bounds–does that sound like “not falsified”?

Ron Cram

Craig,
re: 169

I would prefer scientists stick to conclusions they can support without fear of contradiction. The Douglass et al paper concludes:

We have tested the proposition that greenhouse model simulations and trend observations can be reconciled. Our conclusion is that the present evidence, with the application of a robust statistical test, supports rejection of this proposition.

I would have preferred the conclusion read something like this:

After testing the proposition that greenhouse model simulations and trend observations can be reconciled, we conclude the 22 models we tested have mixed results. Using data currently available and considered appropriate, 19 of the 22 models failed the test and 3 of the models passed.

Certainly the paper would still come under attack from RC, but that says more about RC than it does about the science.

Ron Cram

bender,

The models, even the ones that failed the test, have some value in that they teach us what we do not know about the climate. No matter how much better the models eventually become, I am not convinced they will ever have any predictive value. Scott Armstrong is one of the leading thinkers in scientific forecasts and he points out that climate scientists have not even taken the first step to prove climate is predictable 100 years into the future.

Cliff Huston

I’m still trying to understand the RC/Beaker view on testing consistency of models vs. observations. I would like to try to describe the way I view the problem and hopefully those with stronger statistics background can tell me what’s wrong with my picture.

Starting with just the models, the IPCC says that the best representation of the climate signal is the average of the outputs of the ensemble of GCMs. The ensemble can be viewed as a collection of instruments, with unknown errors (noise, bias, etc.), whose the individual outputs when averaged, largely reduce the unknown errors and produce a good signal.

Given that, the first thing of interest is to test the multi-model average against the individual models to see if any of the models are outliers, that is see if any of the models are inconsistent with the multi-model average. As Beaker has pointed out many times, the correct test is to use two standard deviations as the limit for model data that is consistent with the multi-model average. To my mind, the usefulness of this test is to identify and eliminate any of the individual models that are inconsistent with the multi-model average or at minimum, gain an understanding that the multi-model average is potentially biased by the outliers.

At this point, I have an understanding of the nature of the signal represented by the multi-model average. Now I would like to know the limit of significant values that this signal can have so I can compare it against some other signal. I can’t use two standard deviations as above, because that measure includes errors that cancel in the average that represents the signal. What I can use is two standard errors as the limit, because that measure accounts for the averaging effects. I believe this is correct because I’m no longer concerned with nature of pieces that make up the signal, but rather the nature of the average signal.

Now I can test to see if the multi-model signal is consistent with the observations by looking for overlap between multi-model signal two standard error limits and the observation error bars. This can be done against each observation individually or I could form a multi-observation average, as with the models, and look for overlap with the multi-observation average error bars.

The problem I see with the RC/Beaker approach to testing consistency is that they want to treat the observations as though they were just another model added to the multi-model set and apply the two standard deviations limit to see if the observations are outliers. This make no sense to me, because the observations are not part of the multi-model average, they are separate measures. This makes as much sense as taking Willis’ frog and moose and concluding that they are the same animal, because they share a common gene. The multi-model signal is not the same thing as the observation signal and the point is to compare those separate signals for consistency. Throwing the component parts of the multi-model signal into a blender with the observation signal and declaring consistency because they share a value is, IMHO, just plain silly.

So there you have my view and I would greatly appreciate any corrections where I have it wrong.

Cliff

Kenneth Fritsch

Re: #171

After testing the proposition that greenhouse model simulations and trend observations can be reconciled, we conclude the 22 models we tested have mixed results. Using data currently available and considered appropriate, 19 of the 22 models failed the test and 3 of the models passed.

I would like to see both the naive (see Willis E in Post #16 above) and more complete statistical analysis/tests of those three models that would lead to this conclusion.

I have a major problem with using a climate model output for a region, comparing it to the observed for that region and then failing to test how well that model performs in another region of the globe against observation. I think Douglas makes that error here as I assume the criteria he used to eliminate or qualify the models were restricted to how they performed in the tropics. Fitting a model to a region and restricting evaluation to a region and/or selecting from global models based only on how they perform in that region and ignoring how they perform in other regions provides methods rife for over fitting and data snooping.

I have assumed that the term GHG fingerprint of climate models for tropical surface to troposphere warming trends had to do with the observation that all models show the same biases regardless of how well they agree in other regions of the globe. Claims for climate models skill appears to be more or less relegated to a global mean temperature trend with even the more ardent supporters conceding they have much less skill in getting the regional climates correct. Anyway I can see why the Douglas paper used a mean of the ensemble of models to test the “finger print” against the observed data. When Douglas starts tossing out models without more detailed and comprehensive analyses and suggests that we all do it using our own criteria, he has gone a step or two too far.

When Judith Curry starts tossing out models that do not pass this test (as she states below) on a regional basis, I would want to know much more about how these models perform in other regions. Climate models that might be constructed to look only at a certain region (if that happens) present an even more complicated situation for statistical testing.

This is particularly true if you are using the IPCC results in some sort of regional study. The simulations should pass some simple observational tests: a credible mean value, a credible annual cycle, appropriate magnitude of interannual variability. Toss out the models that don’t pass this test, and look at the projections from those that do pass the test.

Patrick M.

re #176 (Michael Smith):

I think you are on the right track. But rather than picking which models to remove subjectively, you might want to show that certain models are necessary to pass the inconsistency test and then discuss why those models should be removed. It’s only a subtle difference, but it might help avoid the inevitable RC argument that you are cherry picking, (that may be why Douglas didn’t remove any models). It will also shift the burden onto the RC types as to why those models should be included. I don’t think they will be looking forward to defending the inclusion of the “outlier” models.

Andrew

I have yet to see one person denounce Singer’s obviously unsupportable claims regarding this paper.

Done. Shame on the old guy. Now please move on.

steven mosher

re 180. More uncertainty. just what bender asked for. Looking forward to the post

Jon

bender says:

And, Jon, Raven has a question for you in #2.

2: CCSM3
8: ECHAM5
12: MIROC3.2 Merdes

It depends on under what assumptions they’re run, obviously, but averaged under A1B emissions for example CCSM3 is ~+2.8C, and ECHAM5 and MIROC3.2 Merdes are around +3.4C for end of century. Under A2 CCSM3 is +4C pretty much on the nose while the other two are just under.

[Prediction: dodge.]

I’d say your prediction seems to be… inconsistent with observations, wouldn’t you?

Kenneth Fritsch

Re: #165

1. The Douglass et al paper is fair game to be audited just like any other. As such it should be judged on its success or failure to achieve what it claims to.
2. It claimed to show inconsistency when it did not. Whether or not this was an error in understanding the language used or a genuine but mistaken belief that it did, the result is a failure to properly demonstrate the claim.

Interesting, Jon that, while I disagree with your conclusion on the paper that you continue to repeat, an audit being so sensitive to the term inconsistent would have missed its consistent use in the AR4 as noted at CA by Steve M.

By the way I looked for a definition of consistency and inconsistency in statistics books, a statistics glossary and published papers and came up with the following:

Click to access 07Prescher.pdf

A statistical estimator attempts to guess an unknown probability distribution by analyzing a sample from this distribution. One desirable property of an estimator is that its guess is increasingly likely to get arbitrarily close to the actual distribution as the sample size increases. This property is called consistency.

Click to access fukumizu07a.pdf

As in many statistical methods, the target functions defined in the population case are in practice estimated from a finite sample. Thus, the convergence of the estimated functions to the population functions with increasing sample size is very important to justify the method. Since the goal of kernel CCA is to estimate a pair of functions, the convergence should be evaluated in an appropriate functional norm; we thus need tools from functional analysis to characterize the type of convergence.

The purpose of this paper is to rigorously prove the statistical consistency of kernel CCA. In proving the consistency of kernel CCA, we show also the consistency of a pair of functions which may be used as an alternative method for expressing the nonlinear dependence of two variables. The latter method uses the eigenfunctions of a NOrmalized Cross-Covariance Operator, and we call it NOCCO for short.

http://stats.oecd.org/glossary/detail.asp?ID=5125

An estimator is called consistent if it converges in probability to its estimand as sample increases (The International Statistical Institute, “The Oxford Dictionary of Statistical Terms”, edited by Yadolah Dodge, Oxford University Press, 2003).

In other words a common statistical use of the term consistent could deal with the relationship SEM = SD/n^1/2 but would not apply to Beaker’s use of the term inconsistency.

The statistical glossary linked above did not have a definition for inconsistency but did give the definitions below:

Consistency error:

A consistency error refers to the occurrence of the values of two or more data items which do not satisfy some predefined relationship between those data items.

Consistency check:

A consistency check detects whether the value of two or more data items are not in contradiction.

I would have to agree with Cliff that Beaker or perhaps you, Jon, or anyone here needs to produce a published definition of the term inconsistency as it would commonly be apply to statistical testing.

Kenneth Fritsch

Re: #182

I went back to the reference that Beaker gave for showing the use of the term inconsistent with statistical testing. I searched for the word “inconsistency” and give below all the pertinent references in the paper. I replaced the SD for standard deviation as the paper used the Greek letter sigma.

The inconsistency being a measure of 2 SDs just does not come through in these excerpts. Note the use of “moderately inconsistent” with 2.6 and 2.3 SDs and dust is inconsistent with the data at the 94% (1.9 SD). It appears to me that the authors are referencing inconsistency to some changing level of significance and not using inconsistency as a constant measure of significance.

These possibilities are moderately inconsistent with the data at the 99.0% (2.6 SD) and 97.7% (2.3SD) confidence levels, respectively.

This dust is inconsistent with the data at the 94% (1.9 SD) confidence level (although the true inconsistency of this scenario is derived from the product of the two individual likelihoods, i.e., 98% or 2.3 SD).

The distribution of synthetic EB−I values is asymmetric and implies an uncertainty in the measurement for SN 1999Q of +1SD = 0.11 mag and −1SD = 0.17 mag (including the systematic uncertainties from K-corrections and the J-band zeropoint). The results are consistent with no extragalactic reddening and inconsistent with Galactic and gray dust reddening at nearly identical (though marginally higher) confidence levels as the MLCS fits.

The data for SN 1999Q are consistent with no reddening by dust, moderately inconsistent with AV =0.3 mag of gray dust (i.e., graphite dust with minimum size > 0.1 μm; Aguirre 1999a,b) and AV =0.3 mag of Galactic-type dust (Savage & Mathis 1979).

This alternative to an accelerating Universe (see Totani & Kobayashi 1999) is inconsistent with the data at the 99.9% confidence level (3.4 SD).

Click to access 0001384v1.pdf

Ron Cram

re: 177
Kenneth,

The sample conclusion I proposed for the Douglass et al paper was not intended to validate the three models that passed the test. Certainly a number of other tests could be proposed and they may fail any number of them. In his comments above, Douglass writes that three models passed the tests they applied. The paper should have stated that.

I dislike using statistics to make claims about “the models” which may unintentionally mislead readers into thinking the conclusion applied to “all” of the models. I think it far better for researchers not to overstate the conclusions that can be reached from their research.

I think Douglass et al are on the right track and have provided a valuable service. I would like to see a battery of tests applied fairly to all of the models. The purpose would not be to see which of the models would provide the best forecasts for the future, because I do not believe computer models have any predictive value. The goal would be to learn what we don’t know about the way the climate works.

RomanM

#187 Kenneth

I would have to agree with Cliff that Beaker or perhaps you, Jon, or anyone here needs to produce a published definition of the term inconsistency as it would commonly be apply to statistical testing.

This isn’t going to happen because NO such definition exists. The only technical definition of consistent in statistics is the one you found concerning the probabilistic convergence of estimators. Beaker admitted as much in #65 where he states

You won’t find inconsistent defined that was as an individual term, as it has another deeper meaning as a jargon term. However it does have a clear statistical implication , I posted an example of this on the other thread and gave full details of the paper. You can find plenty more by performing a Google Scholar search for “inconsistent with the data”.

It has no specific generally understood meaning in the wider statistical community as jargon nor is there a “clear statistical implication” of its meaning. It means exactly what the authors think that it means – the claims about the relative accuracy of the model results are not in relative agreement with the observed values. Criticism that the authors didn’t understand a specific meaning is simply and clearly unfounded.

Beaker threw out this red herring as part of a justification that the “correct” analysis as accepted by statisticians is to show that the results for EVERY model have to be ridiculously far from the observed tropospheric observations for anyone to be able to claim that the model results are “inconsistent” with reality. In #53, he states

One way of obfuscating with stats is to use phrases with pre-existing statistical interpretations (such as “inconsistent”) to mean something else, when there is already an existing statistical term with exactly the desired meaning (”biased”). The problem I have with the paper is that it makes a very big claim (inconsistency) but performs a test that can demonstrate only bias, a lesser claim (I have lost count of the number of times I have had to type that ;o). If they want to show the ensemble mean is a poor predictor of the observational data, that is a meaningful test of whether the ensemble is useful (although not the only one), and they have made their case. However bias and inconsistency are not the same thing, and they claim inconsistency.

Obfuscating, indeed…. Pre-existing or non-existing? Inconsistency is not a single property – it can take many forms. It can be systematic bias. It can be the presence of higher than or lower than expected variability in observations or calculated results. It can appear in the form of the distribution of the data or in the presence of outliers. To say that since there is a well-defined term for “bias”, you don’t need to refer to it as “inconsistency” is simply wrong. In this particular instance, to my knowledge, the climate science community has not made any claims or offered any specific evidence that any single particular model is THE “accurate” result. In the absence of such evidence they use the averages of the models, calculate error bars for those averages and present those as their best guesses. The authors have basically shown that these averages are NOT consistent with the observed behaviors. To properly evaluate the results of the individual models is more difficult given the lack of information about their parameters and properties provided by the modelers.

As an aside, in #17, beaker states “To test for inconsistency properly, standard practice would be to show that the mean +-2sd error bars of the models don’t overlap with the mean +-2sd error bars of the data.” In other contexts of making multiple comparisons (e.g. Analysis of Variance), comparisons are done by constructing a single error bar for the difference and checking whether zero is in the interval. The test described may be OK for a quick-and-dirty eye-ball check, but the actual probability value for an individual comparison in the normal distribution case would vary from about .005 (when the SDs are equal) to .05 (when one SD is substantially larger or smaller than the other). How can the overall error rate be controlled when multiple comparisons are done? This doesn’t sound right to me.

Steve McIntyre

IPCC AR4 stated the following:

Whereas, on monthly and annual time scales, variations of temperature in the tropics at the surface are amplified aloft in both the MMD simulations and observations by consistent amounts, on longer time scales, simulations of differential tropical warming rates between the surface and the free atmosphere are inconsistent with some observational records. One possible explanation for the discrepancies on multiannual but not shorter time scales is that amplification effects are controlled by different physical mechanisms, but a more probable explanation is that some observational records are contaminated by errors that affect their long-term trends.

Douglass et al stated:

(1) In all cases, radiosonde trends are inconsistent with model trends, except at the surface. (2) In all cases UAH and RSS satellite trends are inconsistent with model trends. (3) The UMD T2 product trend is consistent with model trends. …

Summing up, then, there is a plausible case to be made that the observational trends are completely inconsistent with model trends, except at the surface …

On the whole, the evidence indicates that model trends in the troposphere are very likely inconsistent with observations that indicate that, since 1979, there is no significant long-term amplification factor relative to the surface.

I don’t see any relevant difference between the claim in Douglass et al and the corresponding statement in IPCC AR4. Beaker stated:

If Douglass et al. have defined their own meanings of existing statistical concepts, that are greatly at odds with their established use, it is hardly a surprise that their findings are misunderstood.

As to Beaker’s objection, Douglass et al did not define “their own meanings”. They used the terms in an identical way as IPCC and CCSP and thus did not use the terms in a way that is “greatly at odds with their established use” in climate science. If the use in climate science is different than the use in statistics, that’s a different matter, but one that cannot be laid at the feet of Douglass et al.

steven mosher

Jon,

You write:

“As has been noted by Beaker and myself, whether or not the error was due to intent or accident isn’t really the point. ”

snip

“If Douglass et al wish to submit a comment clarifying their goal in a manner non-contingent upon language that has broader implications than they intend, I think everyone involved would be happy, no? ”

Happy? Happy +- bemused. Did they intend broader implications? That’s a interesting
proposition. How do I test someone’s intention from their words. This is known
as the intentional fallacy. Douglas et all applied a method. They showed the data;
they explained the method. They open both the data and the method to review and
criticism. And, we note, that both the data and the method get the scrunity such a
matter deserves. Good. Contrast that with authors who will not show their data, will
not show their methods and claim that challenges are out of bounds. For example, Phil Jones. So, I am quite happy to join beaker and say that Douglas et al has made
claims that do not stand up. We should thank Douglas et al for supplying enough information to question and perhaps refute their claims. Now, turn to Jones. Data? dr
jones? He wont give it. Methods? Wont reveal them. So simple question John. ( adding the h of course) I agree with beaker. That agreement is a function of beaker being ABLE to audit Douglas.
Jones wont release the data or the methods. Am I wrong to Reject Jones results merely on the basis that he will not release the data or the methods?

In my book I am. His papers, my backside. let the john do its work.

“I’m also interested to hear your (or others’) thoughts on why some of the authors of this paper chose to parade it about in the media as a definitive refutation of current scientific agreement re: attribution of warming, explicitly in regards to policy and viewed in light of past similar claims regarding other regulatory matters.”

I would hope that you have more sense than to speculate about intentions. Definitive refutation? Your words, strawgirl argument. Maybe give your argument a sex
change operation and upgrade it to a strawman.

Bluto.

Cliff Huston

RE:198 Cliff Huston

Sorry, Beaker not Bleaker.

Cliff

M. Simon

Re #194,

I’m also interested to hear your (or others’) thoughts on why some of the authors of this paper chose to parade it about in the media as a definitive refutation of current scientific agreement re: attribution of warming, explicitly in regards to policy and viewed in light of past similar claims regarding other regulatory matters.

It seems to me the correct statement given that there seems to be considerable disagreement. Not to mention the newly found Childe Cycle (not previously included in the models).

a definitive refutation of current scientific agreement

should be:

a definitive refutation of current agreement among some scientists

In fact can we really say anything useful about these models unless we know they included the Childe Cycle?

With apologies to that Great Scientist Gordon R. Dickson.

Eggplant fan

The page http://www.gisclimatechange.org/faqPage.do includes the statement “CCSM modelers may perform additional runs where they supply ocean observational data, which forces the system to incorporate interactions between ocean and atmosphere and pick out specific events like El Nino.” which indicates that they could do what I suggested in #202. It also includes the statement “Essentially, the earth’s climate can be considered to be a special ensemble that consists of only one member.” That would seem to be consistent with the RC point of view as noted by Steve McIntyre in #52 (although not addressing the problem of averaging across models). It is difficult to tell then exactly what the IPCC means by “in better agreement with observations”.

beaker

Good morning all, there has been a lot of posts since Friday, and I’ll attempt to address all the comments that seem to be asking for a response from me, but they will have to be brief due to their number. If you feel I haven’t adequately addressed the point, just say so and I’ll have another go.

Steve McIntyre #93: 16 in your first innings sounds pretty good, four boundaries suggests you have a good eye, it is well above two standard errors of my league batting average (but not two standard deviations ;o).

Pat Keating #95: I have always thought the frequentist perspective rather odd as the intuitive interpretation of a probability is not a long run frequency in most cases. A frequentist can give no direct answer to the question “what is the probability that this hypothesis is correct?”, but it is the question that we would most like answered. At least a Bayesian can discuss whether it is more probable than an alternative hypothesis. See also PIs apposite comments in #97.

Steve McIntyre #96: A rewording of the point in the paper would indeed be possible, to say that “the ensemble mean is inconsistent with the data” would be a good start. From a statistical point of view it seems like a slightly un-natural way of putting it, but it makes the point that it is only the mean that is shown to be inconsistent, rather than the models themselves.

Phil #98: I was playing rather than watching (Saturday and Sunday and paying for it now, I ache all over ;o). I am also looking forward to the ENG-NZ series.

Kenneth Fritsch #102

I can only ponder why Beaker who has statistical insights to offer here keeps going back to the word “inconsistent” used in the Douglas paper.

Because my statistical insight is that there is a big difference between “the models are inconsistent with the data” and “the ensemble mean is inconsistent with the data” (see my coment to #96). Douglass claim the former, but only establish the latter. As for pondering motives, I believe a scientist should always assume good motives and address the argument directly rather than question the source. I am happy with the idea that the problem with the Douglass paper is unintentional, and that they did not appreciate their wording encourages misinterpretation.

Cliff Huston #104 : Read the summary of Douglass et al. they make no mention of the ensemble mean, but only talk of model trends.

N Taylor #107: No, it is not only of philosphy issue. There is a real difference between “the models are inconsistent with the data” and “the ensemble mean is inconsistent with the data” (thanks Steve for the compromise wording!). Baysian stats has very sound theoretical foundations, see the works of RT Cox, Harold Jeffreys and ET Jaynes. See also the (again) apposite comments from PI in #112.

As it happens, I do a lot of reviewing of medical papers based on the particular statistical methods with which I am most familiar, and I see many papers with very poor statistical evaluation. It is a field where a blog systematically aduiting problematic papers would be very valuable. Having seen the stats in some cardiology papers it is a great spur to exercise ;o).

Pat Keating #109

the language Beaker used (based on “the Baysian perspective”) stated that the observational data was “implausible”, a truly topsy-turvy viewpoint.

Frequentists statisticians do much the same thing in maximum likelihood based approaches, it isn’t a solely Baysian view. However, it is clear that I need to adjust my language to be a little less statistical in order for my arguments to be understood!

Steve McIntyre #110: I too am keen to reach an agreement on the issue. The compromise wording that you suggested in #96 seems helpful. The issue is the difference between the ensemble mean being inconsistent and the models or the ensemble being inconsistent.

Cliff Huston #111: There is a difference between the inconsistency of the ensemble mean and the inconsistency of the ensemble itself. Do the IPCC recommend the use of the ensemble mean without error bars? If so that is a practice that needs auditing and adds value to the findings of Douglass et al (although it is only implied). If the IPCC suggest the mean with the +-2sd error bars, then it is wrong to apply the +-2se test.

Jon #114: The SEPP press release goes a long way to encouraging the (mis)interpretation of the paper exemplified by the RC article. If the model ensemble itself were shown to be inconsistent, that might be a reasonable basis for the argument that AGW arguments based on the models were unsustainable, but not “inconsistency” in the +-2SE sense establised by Douglass et al. However even then it would not be evidence that AGW wasn’t happening, just that arguments should not be based on the models alone (but caveat emptor – I am not a climatologist!).

Lance #115: If the models had failed the +-2SD they would have been “slain” (at the 95% level of confidence), you acknowledge that they haven’t been “slain” by Douglass et al. That shows that the difference is not merely semantic, but substantial.

Bob B #116: Do the models claim to predict future temperature with any certainty? The +-2SD error bars give an indication of the certainty, and if they are wide then they are not claiming certainty. That is why IMHO the RC response makes a much better job of highlighting the problem the models have with tropical trends, the error bars are very wide and the data only overlap on the fringes. All predictions should have error bars and they should be used, not ignored.

Willis Eschenbach #117: The +-2SD error bars are very likely to be wider than the range of the data if you only have four datapoints. Why not just plot them and see. But first, you need to include the stochastic uncertainty by averaging over all runs and use the original data without normalising to the surface trend. The variation at the surface is part of the uncertainty of the ensemble, so it is inappropriate to arbitrarily delete it. This means you still have not performed the test properly.

NTaylor #122: For statistical validation of statistical methods, the thing that comes closest to that would probably be “computational learning theory” which covers the analysis of leaning algorithms (statistical models), see e.g. VC (Vapnik-Chervonenkis) theory the PAC framework (probably approximately correct). Saying that frequentist and Bayesian approaches have similarly strong theoretical foundations seems a good summary. However the Bayesian definition of a probability is closer to the intuitive interpretation and consistent (in the usual sense ;o) when a long run frequency is meaningful. Frequentists methods are preferable though where formal experimental design is possible (but I don’t generally work on such problems, so that is just my opinion).

Bender #125:

Don’t let beaker off the hook yet.
1. He has not explained why he thinks there is a bias in some of the GCMs.
2. He has not expained if tropical tropospheric warming is or is not part of the AGW fingerprint.
3. He hasn’t explained why GMT has flattened since 1998.
He’s half-answering all the easy questions, dodging the important ones.

I was quite amused by this post, it should be obvious that I am a statistician, not a climatologist and not a climate modeller. Asking me for authoritative answers to any of those questions is a bit like hiring a concert pianist in the expectation that he can fix the transmission in your car! ;o)

I’ll have a go at 3 (but I am just repeating arguments gven elsewhere, with which I am sure you will be familiar, however they seem reasonable to me). The magnitude of any AGW trend is small in relation to the short term variability due to e.g. ENSO, so over a ten year trend the short term variability can obscure the true trend (especially if you bias it by arbitrarily choosing to start at a conspicuous peak, just a guess, but maybe you would get a different result if you started the window in 1999?).

Bender #134: If you look, you will find that your recipe is equivalent to the G^2CM idea I mentioned in #13. The ensemble we implicitly have is a small, ad-hoc implementation of the same basic approach. I am not sure why the ratio you give is an indication of whether the models are “crap”, but the methodology is much closer to the RC article than the Douglass paper (which explicitly ignores these uncertainties by concentrating only on the ensemble mean).

kuhnkat #139: It is not so much the word “inconsistent” as its use to describe the results of a hypothesis test to make phrases like “the models are inconsistent with the data” when the test only shows the ensemble mean is “inconsistent” withe the data (thanks again Steve). Dr (Prof?) Douglass has already sent in one clarification in response to criticisms raised, my criticism could easily be resolved in a similar manner, and I think this would be beneficial. The main advantage of the ensemble is that it give an indication of the uncertainties involved in the prediction around the expected value.

Steve McIntyre: If the word “inconsistent” is used to descibe the results of a statistical hypothesis test, it is reasonable to interpret it in a statistical manner, if it is in a report making qualitative statements it is reasonable to interpret it in a more colloquial way. Anyway, your compromise wording clarifies the point rather better, in a statistical sense “the models are inconsistent with the data” and “the model means are inconsistent with the data” are very different. Also I am more comfortable with loose statements being made that unintentionally overplay factors pointing out the limitations of the authors arguments than those that unintentionally overplay the strength of their argument, however not all may agree with that.

Joe Crawford #163: You are quite right, I should have said “If a number of groups independently come up with similar models, it is an indication that a particular set of assumptions is [considered] more plausible than others [by the climate modelling community]”.

Craig Loehle #170: That is why the RealClimate article did a better job of pointing out the problems with models than Douglass et al. If the observations are within the error bars than the data are plausible given the assumptions of the models, in which case the models while not being shown to be right have not been shown to be wrong either. As I said, consistency and usefulness are not the same thing (even unbiasedness is not a guarantee of usefulness).

Ron Cram #171: A very good point. If you need to make a claim that is likely to be contentious, it is a good idea to adopt the strategy used in chess and make the move that minimises your opponents maximum gain. It is true that you then procede only via small steps, but they have the advantage of being in a uniformly forward direction!

Patrick M 179: The reason that all of the models should be in the ensemble is that all of them are needed to reflect the systematic and stochastic uncertaintiesthat affect their predictions. By taking models away, you make the ensemble error bars look as if there is more confidence in the model output that can is actually warranted.

Steve McIntyre #180: IIRC the Santer paper referenced in Douglass et al draws the conclusion that the “inconsistency” is due to problems with the data rather than the models. So we have the modellers (and the RAOBCORE group) saying it is a problem with the data and the data people (I believe that is one of Prof. Christies specialities at least) blaming the models. So who do we believe? Well I for one am in no position to decide, so I will adopt a true Bayesian stance and assume there is some truth in both positions (i.e. keep an open mind). ;o)

Cliff Huston #182: The question answered by the RC +-2SD question is “are the observations plausible given the systematic and stochastic uncertainties involved”. That is the relevant question for falsification of the models that might invalidate AGW arguments based solely on the models. The problem is a disconect between the test they performed and the summarisation of the results.

Kenneth Fritsch #187: I did say that there is a deeper meaning to “inconsistent” as a term (rather then in its use in the phrase “the models are inconsistent with the data”), and you have found it. It isn’t relevant to this discussion in any way.

Kenneth Fritsch #189: You are missing the point, in the paper I referenced, the word “inconsistent” is used with respect to tests based on the standard deviation not the standard error of the mean, that is the issue. I have already explained that the results of a statistical test are given at a confidence level that dictates the number of standard deviation/SEMs that are used, that is not the point I am making.

Steve McIntyre: Do the IPCC reports make these statements informally, or are they descibing the results of specific hypothesis tests? It is the context that decides whether a statistical or informal interpretation is required. It seems to me that the IPCC are being overly cautious in thier statement (although as the models are not far from being inconsistent in the statistical sense anyway).

Cliff Huston #198: I am looking into the IPCC reasons for the use of the ensemble mean, as a result of Steves post, but I would prefer not to comment directly until I have read up on the subject to make sure I understand the relevant issues. If they are saying that you should use the ensemble mean in isolation, or in conjunction with SEM error bars, then an audit is certainly warranted. If they suggest the use of the mean with SD based error bars, then that seems reasonable (but not as good as using the ensemble itself). However if they say to use the SD error bars, it is even more unreasonable to use the SEM in validation execrises.

Well that is more than enough typing for the moment, I’ll be back with some more substantive posts later.

beaker

Having started to read up on the IPCCs reasons for the use of the ensemble mean (which also are possibly a fair target for an audit) I am now fairly sure that the statistical test performed in Douglass et al. is meaningless (although the point they seem to be trying to make with it, that the models do not adequately reproduce the observations, seems perfectly valid). My reasoning is as follows (appologies if I have misused climate modelling terms, as is quite likely, I hope my point can still be discerned):

The ensemble consists of a variety of models, which are intended to represent the systematic uncertainties, i.e. the modellers uncertainties over the detailed physics governing the climate. Some models are represented by more than one run, such that the spread of runs for a model attempts to cover the stochastic uncertainty (i.e. the variability that is observed from a chaotic system depending on the initial conditions). So what does the mean of a multi-model, multi-run ensemble actually mean? As far as I can see, averaging over models and over runs attempts to average out the systematic and stochastic uncertainties, leaving you with an estimate of only the “forced component” of the climate, after the effects of the initial conditions have been averaged out.

Now lets consider the observations. We can consider the Earth as being a climate simulator, where the systematic uncertainty is zero (the physics is of course exactly right), but we have only a single run (the observed reality). The observed climate therefore does depend on a single set of initial conditions.

This means that even if the ensemble mean met its design brief perfectly, you would not expect it to exactly match the observed data. The ensemble mean aims to model the mean climate, averaged over all plausible initial conditions, whereas the observed climate is the climate observed for one particular set of initial conditions. There is no reason to expect them to be identical.

Why does this matter? Because even if the ensemble mean gave a perfect representation of the mean climate, there would be a point where the ensemble was large enough to fail the 2SE test, even if it were correct. That is clearly absurd, which suggests that it is the wrong test, basically it is not a like for like comparison, the ensemble mean represents only the forced component, the observed data represent both the forced component and a component due to the initial conditions.

What the test does tell us is that the true mean trend of some implicit distribution of models differs from the observed trend, and that we can be 95% sure that the observed difference can’t be explained as an artifact of the particular sample of models forming this particular ensemble. The strongest statement about the models that can be justified by this is that for the models to give a good estimate of the forced component, we would require that the difference can be explained by the variability of the climate due to the inital conditions. Can we make arguments for the plausible magnitude of this variability without recourse to the models (and therefore a circular argument)? I don’t know.

On the other hand, it is clearly interesting that the models only pass the conventional consistency test with a D- (as Patrick M puts it in #88), even taking more of the uncertainties into account. That seems to me to be a much more effective criticism of the models (or potentially the data, c.f. the Santer paper cited by Douglass et al.).

beaker

Another thought experiment that makes me question the value of the +-2SE test. Consider an ensemble that was in good subjective ageement with the observed trends, say the difference was of the order of 0.05 K/decade, but that the ensemble was large enough that the observations are only just within the +-2SE error bars, so it only just passes the Douglass test. Now we add two new models that straddle the existing mean in such a way that both the mean and standard deviation of the ensemble remains the same. However, because of the 1/sqrt(n) factor, the ensemble now fails the +-2SE test! Can you justify how it could be reasonable for an ensemble can go from being “consistent with the data” to being “inconsistent with the data”, without the mean or standard deviation having changed?

Again, the aim of the Douglas paper seems reasonable (if not that novel), but the test they use does not make the point.

Michael Smith

Can you justify how it could be reasonable for an ensemble can go from being “consistent with the data” to being “inconsistent with the data”, without the mean or standard deviation having changed?

As a layman, my interpretation would be that as a result of the additional information gained by adding the two new models, we are now that much more certain of the true value of the mean . Hence, the observed trends that previously could be said to be within the 95% CI of the mean — which we have somewhat arbitrarily defined as “consistent with the data” — are now only within some lesser % CI, which doesn‘t meet our standard for “consistent with“.

What I find far more difficult to “justify how it could be reasonable” is the notion that — in a test that purports to compare AGW models versus observations — it is proper to include models that don’t predict significant amplification of warming in the troposphere and models whose surface predictions are more than two standard deviations away from the surface trends — and then use the large standard deviation created by the inclusion of those models to conclude that “there is no clear model-data discrepancy”.

beaker

Michael Smith says:

As a layman, my interpretation would be that as a result of the additional
information gained by adding the two new models, we are now that much more certain of the true value of the mean.

But if the mean and standard deviation remain the same, the (Gaussian approximation of the) ensemble is giving exactly the same hindcast of the tropical trend as it was before. If the hindcast is the same, why does one come in for criticism and not the other?

I think it is relevant to ask what you mean by the “true value of the mean” in the first place. Is the ensemble a i.i.d. sample from some underlying distribution of models? As others have noted this is debatable (especially from a frequentist perspective).

What I find far more difficult to “justify how it could be reasonable” is the notion that — in a test that purports to compare AGW models versus observations — it is proper to include models that don’t predict significant amplification of warming in the troposphere and models whose surface predictions are more than two standard deviations away from the surface trends — and then use the large standard deviation created by the inclusion of those models to conclude that “there is no clear model-data discrepancy”.

The output of the models represents what is considered feasible by the modelling community, given the known systematic and stochastic uncertainties. It just gives the consequences of a set of assumptions, nothing more. If you prune the ensemble, then you reduce (incorrectly) the apparent uncertainty in the models, and make them look as if they are more confident than originally claimed. That is why they should not be pruned.

I have lost track of the number of times I have pointed this out, but the +-2SD test is only interesting if the models fail it. If they pass, it is not an indication that the models are good or even useful. The Douglass test doesn’t tell you if the models are useful either; as I have already pointed out, even if the GCMs had the physics exactly right they would still fail the +-2SE test! The fact that you could have two functionally identical ensembles, where one passes and one fails shows that the test is inappropriate in establishing usefulness or validity.

beaker

Steve #212: The issues are indeed fun to discuss, as the length of #205 demonstrates! Your compromise wording in #96 goes a long way in clarifying the problem I (and others) have with the Douglass et al. paper.

My views on the appropriateness/adequacy of the +-2SE have sharpened rather, the arguments in #207 and #209 seem to make the point rather better than I had previously. However, this doesn’t mean I have a problem with what I take to be the aim of the paper, namely to demonstrate that the observations were not in close accord with the models. The really interesting question would be whether this is a problem with the models or the data or both.

I think part of the problem lies in treating the models as if they were statistical models (e.g. linear regression) explicitly fitted to some data, which they are not.
It might be a good idea to have a look at the ensemble issue, I use them routinely in my work (though not ensembles of GCMs), perhaps we should discuss it off-line via email?

Eggplant fan

Geoff Sherrington #204

I guess I’ve come across as far more negative toward the Douglass et al. paper than I intended. The paper does the best possible test against the claims of the IPCC taken at face value: The mean of the different models converges to the observations. If the hypothesis is that different models give a normal/Gaussian distribution around the actual observation, then the T-test (with standard error of the mean) is the appropriate test. RC jumps on the fact the model builders don’t make the same claim (note that the FAQ page I linked in #203 evades the issue of averaging different models altogether), but then goes off on a red herring that is compatible with neither the IPCC nor the model builders’ claims. I fully agree that the Douglass et al. paper shows that there are serious concerns about the modeled results. I find it troubling that the models have not been run with the observed El Nino’s incorporated (which the FAQ indicates is possible) to see what specific effects it has on other observables for a more direct comparison. As it stands now, there is so much uncertainty allowed that it is almost impossible to invalidate the “average-of-the-models model”, which makes it essentially useless for projections.

Kenneth Fritsch

Beaker, you have failed to show that the term “inconsistent” has any accepted and specific meaning in statistics. You keep changing the depth of your views on the use of SEM in the Douglas paper. There may be good reasons for not treating the model outputs as one would, for example, a samplings for control charting but I do not believe that you have made a good case for why this could not be done as a first approximation as Douglas et al. have done.

Surely one cannot consider the observed data as another result at an equal level with the model outputs because it is the target and therefore it is entirely wrong to attempt a comparison with 2 SD or a range of model outputs as was done in some earlier papers as I noted in previous post. One hardly needs a thought experiment to show that including model output outliers and/or physically unrealistic model outputs can make a range or 2 SD comparison of little value. If one wants to at least narrow the consequences of these problems one would use a comparison to the model mean and to a first approximation using SE seems most practical.

Therefore a choice between the Gavin/Karl et al. and the Douglas et al. approaches for comparing model out puts to observed data easily goes to Douglas. In fact the use of ranges and SD are fatally flawed in my estimation.

beaker

I forgot to add, the ensemble probably under-estimates the true systematic uncertainty of the models, as the modellers are likely to have used only what they consider to be the most likely physics, rather than sampling model parameters from the range they consider plausible.

beaker

Kenneth Fritsch #219: There is a good reason why I change the depth of my view in the use of SEM in Douglas et al. It is because I listen to the views expressed by my “opponents” in the debate (they are not really opponents as we are auditing), and if I see value in them I moderate my opinion. How Bayesian is that? ;o)

I also think about it and try to construct illustrative examples that determine whether the test makes sense, and revise my opinion accordingly. Isn’t this what a scientist should be doing?

However, I have now demonstrated that a perfect model can only pass the test in the assymptotic case (i.e. as the number of models becomes large) by a fluke. For me that shows unequivocally that the test is wrong. If you can show me that my reasoning is faulty, I will revise my opinion, demonstrating an open mind.

I have also shown that functionally identical ensembles don’t neccessarily give the same result for the test. How can the test evaluate the usefulness of the model if it doesn’t depend on what the model actually says?

Kenneth Fritsch

Re: #

The Douglass test doesn’t tell you if the models are useful either; as I have already pointed out, even if the GCMs had the physics exactly right they would still fail the +-2SE test! The fact that you could have two functionally identical ensembles, where one passes and one fails shows that the test is inappropriate in establishing usefulness or validity.

Please demonstrate in detail how the GCMs could have it exactly “right” and fail a +/- 2SE test. It seems you are making a case for using an average of models and model runs and then simply waving off the use of SE in making a comparison.

Christopher

Re: 224
From the abstract/summary

“Model results and observed temperature trends are in disagreement in most of the tropical troposphere, being separated by more than twice the uncertainty of the model mean.”

beaker, could you list your top objections to the paper? I’ve been trying to follow this but I’m getting slowly lost in all the back and forth. Fwiw, I can understand your issue wrt SE vs SD but Douglass lays out exactly what he does, defines uncertainty as 2SE, and then does his analysis. Part of me thinks the point is being missed. The observed ~ predicted agreement is lacking. Could the paper have been improved somehow? Of course, I’ve never published a single thing where I thought otherwise. But is that the issue now? I’ve trying to see this here but feel like some things are being lost in semantics.

Michael Smith

However, I have now demonstrated that a perfect model can only pass the test in the assymptotic case (i.e. as the number of models becomes large) by a fluke. For me that shows unequivocally that the test is wrong.

I see two problems with this argument. First, Douglass is not addressing the asymptotic case. Second, a perfect model will not be evaluated as part of an ensemble; it will be evaluated by itself. (As I understand it, we are using an ensemble precisely because we don’t have a perfect model.) So I don’t see the relevance of your point to the case at hand.

Kenneth Fritsch

Re: #224

Read the summary of Douglas et al. They make no mention of means, they only talk of reconciling the models with the data. If they said the means of the ensemble were “inconsistent” the problem would largely go away. However, as I have shown, that is meaningless anyway.

From the Douglas paper we have:

Evaluating the extent of agreement between models and observations Our results indicate the following, using the 2σSE criterion of consistency: (1) In all cases, radiosonde trends are inconsistent with model trends, except at the surface.

(2) In all cases UAH and RSS satellite trends are inconsistent with model trends. (3) The UMD T2 product trend is consistent with model trends.

We have tested the proposition that greenhouse model simulations and trend observations can be reconciled. Our conclusion is that the present evidence, with the application of a robust statistical test, supports rejection of this proposition.

Models are very consistent, as this article demonstrates, in showing a significant difference between surface and tropospheric trends, with tropospheric temperature trends warming faster than the surface. What is new in this article is the determination of a very robust estimate of the magnitude of the model trends at each atmospheric layer. These are compared with several equally robust updated estimates of trends from observations which disagree with trends from the models.

The authors are very obviously referring to the ensemble of models and the averages, not only by the wording in these excerpts, but by the statistical testing that they carried out.

I do not find putting words and meaning into the paper that are obviously not there very instructive.

You have made several declarations about the physical and stochastic nature of the model outputs without showing in detail how they would specifically affect the statistical testing of the model output in this case. I am more familiar with analysis at CA using the data in a paper to demonstrate a point. In my mind you are throwing caveats into the equation that are already accounted for in the assumptions of the tests.

beaker

Michael Smith #229: I mentioned the assymptotic case simply because the failure of perfect model is almost certain in that case. Depending on the apparently unquantifiable “sigma of the climate giving the forcing”, the perfect model will fail with finite (potentially quite small) ensemble sizes, the larger the ensemble, the more likely it is to fail. Your second comment shows why you have misunderstood this. Even a perfect model will have stochastic uncertainty, it is inherent in chaotic systems, so climatologists would still use an ensemble even if they had a perfect model.

Kenneth Fritsch

Beaker you referred to GCMs getting it right not GCM. Regardless it is clear that Douglas et al. were comparing an average ofensemble of model outputs. And outputs that were rather consistent in Ts to Tt trends.

None of the model outputs matched the observed, but if you want to use (conveniently) here an average (that you imply has no meaning as there is only one perfect or near perfect model) to a 2 SD confindence limit to say that such a model could exist by less than mere chance, you might want to check your premises in arguing against the use SE in a comparison.

beaker

Roman M: O.K., in that case if we were to use the two ensembles to decide policy, how would the resulting policies differ? They wouldn’t, because both ensembles tell you the same thing. Think of it like this, if we had an infinite number of models in the ensemble, does this mean that the ensemble insists that the observed climate should be identical to the ensemble mean (as the SEM is zero)? No of course not, because the standard deviation gives the uncertainty of the ensemble, not the standard error of the mean.

I think either the IPCC recommendations regarding the ensemble mean is badly flawed or it is being badly misinterpreted!

RomanM

#237

Deciding that two things are not the same (the focus of the paper) is not necessarily what a policy decision would be based on. In practice, for such decisions, one can form an alternative hypothesis that the difference exceeds a given (separately determined on practical importance consideration) amount. You still have the statistical problem of deciding whether this is indeed the case based on the uncertainty in the sample, the observed sample values, and the sample information, i.e. the sample size. In such a case, even your infinite sample could end up not rejecting the null hypothesis if the difference is less than or equal to the predetermined limit. In the case as well, both ensembles do not tell you the identical thing.

RomanM

#238

What do you mean by “clone” the model in each ensemble? If that means writing the result of a single run down twice, then you have not increased the sample size and you have no new information. If you run each model twice, and get the same result in each case, I would conclude all of the models are basically of zero variablility (i.e., deterministic) and that doing statistics with them is pretty much a moot exercise. If the runs are different then now I do have new information and the ensembles are different.

RomanM

#240 Jon:

I apologize. You are correct, my response was phrased in somewhat disparaging terms aimed not directly at you, but at the concept of protecting the sanctity of the models by putting any serious evaluation of their results out of practical reach. Virtually every dire consequence predicted by the AGW advocates is based on models. Without them, there is a very little real evidence of how important the A is the recent GW. And, yes, I care very much whether they are correct since there are those advocating incredible changes based on these models.

There ARE persons so imbued with an AGW agenda that I would not trust them to embellish results by suppressing contrary information. Try reading the posts on the RC site. Any implication that anything (major or trivial) could be questionable is quickly and firmly beaten down by language far worse than what I wrote. You will excuse me if I prefer to ignore the self-protective put-real-tests-out-of-reach admonitions that you quote.

Michael Smith

beaker, in 233 wrote:

Even a perfect model will have stochastic uncertainty, it is inherent in chaotic systems, so climatologists would still use an ensemble even if they had a perfect model.

Okay, but it is not going to be the same ensemble we have now, is it?

Jon

Should have read:

And there will be a lot of crawling before walking before running, I would imagine.

beaker

Michael Smith #245: Well if you want to defend a test that will reject even a perfect (true) model, I would venture you are setting your standards a little high, to say the least! ;o).

Note that a climatologist (assuming he understood the benefit of an ensemble) would include as many models as computationally feasible to account for as much of the stochastic uncertainty as possible. So best practice would lead to the model being more likely to fail the test than bad practice!

DeWitt Payne

If the SEM of the ensembles is a function of 1/sqrt(n) and n becomes very large then isn’t the consistency test with the standard error of the data? Aren’t there time series methods for deriving an estimate of the data variability that take into account persistence and autocorrelation or whatever?

beaker

Kenneth Fritsch #252: The benefit of averaging over the ensemble is that it reduces the effects of the stochastic uncertainty. However, this does not mean that the ensemble mean will be a close match for the observed trend, even if you use the true model. The reason for this is because the ensemble mean only gives the forced component of the climate. However the observed trend is due to both this forced component and a component that depends on the initial conditions. Whether the ensemble mean is close to the observed trend depends on the strength of the component that depends on the initial conditions. That is what Steve terms “the sigma of the climate given the forcing”, which unfortunately we don’t actually seem to know.
You have to distinguish between the uncertainty of the ensemble and the uncertainty of the mean of the ensemble. The SE doesn’t give a useful indication of the claimed uncertainty of the ensemble, and so should not be used for validation. I’m sorry, I seem to have run out of ways of trying to explain this.

I understand the aim of the Douglass paper, but the test that they use didn’t make that point as it treats the ensemble mean as a confident prediction of the trend, with the SE as the error bars. It isn’t the SD is the error bars for the ensemble uncertainty. The SE is only the error bars for the uncertainty in estimating the mean of the underlying population of models based on a finite sample.

You have misunderstood my argument. The SE test will be more likely to reject a large ensemble based on the true model than a single run, because of the 1/sqrt(n) factor!

beaker

Kenneth #254: We should not be testing whether the mean of the ensemble is different from the mean observed trend. I have already given an explanation of why it is unreasonable to expect this to be zero (I repeted it for your benefit in #253). When you understand why this is not a reasonable expectation, you will understand why even the true model will ultimately fail the 2SE test.

I can’t put it any clearer than this. There is no reason to expect the difference between the observed trends and the ensemble mean of the perfect model to be
zero. Given that, why test for the difference of the two means being zero?

Ron Cram

beaker,

In his comments above, David Douglass wrote:

Invent your own tests or criteria. The main conclusion will be the same – only a few models can be reconciled with the observations.

Do you agree or disagree with this statement?

Kenneth Fritsch

Given that, why test for the difference of the two means being zero?

I added an E (it could have a +/- E) in the difference in X1 – X2 that perhaps you missed.

You also make a big leap by declaring that the “natural variation” of the component in the models that can be averaged out is in the observed data to a significant extent. The observed (and modeled) values in Douglas are dealing with differences (or one could use ratios) and one would be required to show that that difference leaves a significant residual in the observed that is averaged out in the model results. We are also comparing trends that surely smooth out the low frequency natural variations in the observed.

I recall some relationships that David Smith showed on one of these threads that had a good annual correlation over the satellite era between Ts and Tt in the tropical zone for the observed data. I will attempt to find it again.

Of course when E for natural effects is shown to be large compared to the forced effects we have another debate on our hands.

steven mosher

Kenneth,

I think the issue of SEM and SD is played out. Besides, it not the real issue.

The models are not “beaker inconsistent” with the data. Next tuesday is not “beaker inconsistent” with the data. The key take away is this. The models are all over the map as we saw, the data is all over the map and being “adjusted” by the data chiropractors. The real question is model SKILL.that’s what we really care about. Well, I’m a luke warmer,
So I believe in AGW physics, but I dont trust the models farther than I can pitch
a leather couch.

Hansen claims, for example, that we cannot exceed 450ppm or greenland will do the wicked witch of the west dance and melt. Where is this level of accuracy in prediction supported? where? In models that produce a spread like you see in douglas.? “I got
some quatrains to write,” says moshtradamous.

its been a fun discussion. I’m glad beaker’s hung around,

Oh well. see ya round.

John F. Pittman

From the IPCC #260 W.E. quote

The multi-model ensemble mean estimate of forced climate change is {T} = TF + {Tm} + {T”} where the natural variability again averages to zero for a large enough ensemble.

emphasis mine

Isn’t what the IPCC is saying is that the signal ( forced climate change – the A part of AGW) is estimated from a large enough ensemble mean such that the natural (assumed unforced wrt to the A part and natural forcings) variability averages to zero? From their definitions of natural and forced, are the IPCC are claiming to

reduce the expected error to zero

(from beaker 262), for natural variability and the systemic error to be small?

Kenneth Fritsch

Re: #262

The reason that the ensemble mean of the ideal model will mot be equal to the observed trend is becuase of what Steve termed the “sigma of the climate given the forcing” which affects the observed trend and is unpredictable, so the models can do nothing about it.

Beaker, the “sigma of the climate” is in the models or at least should be and could tend to average out. The sigma of climate is in the observed overall tempersture signal but in this case we are looking at differences in trends. Please explain how that leaves an added sigma of the climate in the observed that is significantly different in the model out put?

We are looking for an effect of GHG forcing on both the observed and models by using this difference in order to zero out other effects. Please explain where the sigma due to non-forcing effects comes into play. And if you can show it in general then explain how it could be estimated vis a vis a significant model to observed difference.

Ron Cram

beaker,

After you answer Willis’s question, please reply to mine in #257.

Phil.

Re #268

Considering those series are derived from the same sensors such a good correlation isn’t surprising:
TLT=Temperature Lower Troposphere MSU 2 and AMSU 5
TMT=Temperature Middle Troposphere MSU 2 and AMSU 5

There’s considerable overlap of the signatures:

srp

Beaker’s analysis of the infinite ensemble has two problems:

1) Everyone would embrace the models if it were proved statistically that they converged to a value that was off from the data but very close to it. For policy purposes, using a reasonable loss function, a model that is known to be wrong but only by a tiny bit is better than one which is not known to be wrong but gives answers that may be very far from the truth much of the time. Bohr’s model of the hydrogen atom is known to be wrong when you look at details of the spectra, but for many practical purposes it’s still a damn good model. Ditto Newtonian physics. You don’t want to confuse statistical significance with practical importance.

2) The whole point of the IPCC exercise is that temperature variablility conditional on the forcing is just transient weather, not persistent climate. Under this maintained (by the IPCC) hypothesis, it is not true that the initial conditions allow a wide range of data trajectories for the means of things like the troposphere/surface temperature ratio. Their hypothesis is that if you wind up the climate and let it go with the same forcings, the averages will converge to the same predictable values; all the sensitive dependence on initial conditions is about weather, not climate. So it is incorrect to suppose that a test of this hypothesis should treat the observed climate statistics as coming from an urn with a widely varying population. The whole point of the IPCC exercise is that when you look at long-run means, the data should come out the same every time.

Kenneth Fritsch

Re: #272

It is simply a month by month plot of the UAH TTL (lower troposhere temperatures) versus UAH TMT (middle troposphere temperatures) for the tropics over the time period from Dec 1978 to March 2008. I need to point out that it is the tropics as that information was left out of the graph title.

Ron Cram

Kenneth,

I’m sorry. I should have been more specific. I do not believe the plot is in degrees C. Is it measured as the anomaly to the monthly average for the surface temp? Or is it something else? As a scatter plot, the trend line cannot be time dependent. What does the trend line represent exactly? Just the fact that when Lower Trop temps are higher, so are the Mid Trop temps? And vice versa? The trend line is at 45 degrees (or would be if the rectangles were squares), what does this indicate to you? What would it indicate if the trend line was at 60 degree slope? Or 30 degree slope? What conclusion do you draw from the plot and why?

Michael Smith

Beaker, you wrote in 276:

Even a perfect model cannot predict the second component and so will fail the test if a large enough ensemble is made, which is absurd.

But in reality, that’s not going to happen, is it?

If the purpose of the ensemble is to account for stochastic uncertainty around the perfect model, wouldn’t that mean the ensemble would only include models that increased the standard deviation?

Kenneth Fritsch

Beaker, your reply is much too general and dismissive to be of much use to me. You will have to explain the “sigma of the climate” and how that sigma is left in the ratios of trends (or R as Douglas et al. terms it in their paper) of the troposphere to surface GHG finger print in the observed data.

From my readings I know that the climate models can model the climate without the forcings included and this will give (or theorectically should give) in each individual run a sigma of the climate over a period of time. We can then take these individual model results without forcings and look at the trends. We can further do the same for the tropical surface and tropospheric temperature trends without forcings.

Since I have seen climate models run without forcings and yielding little or no trends over relatively long time periods (without doing any ratios) I suspect that the climate sigma you refer to would be very small in any trend ratios (as in Douglas) over appreciable recent time periods as derived from climate models or in the observed data. Now if you were to have disagreements with those model findings on a climate sigma as it effects trends then I suspect the models simply are not at such a state of advancement to allow reasonable evaluation of them or even parts of them.

The observed Ts/Tt ratios of trends should (and particularly one that gives a GHG finger print) be relatively free of any climate sigma no matter how manifested or related to a supposed chaotic system’s initial conditions.

Kenneth Fritsch

Ron Cram:

Is it measured as the anomaly to the monthly average for the surface temp? Or is it something else? As a scatter plot, the trend line cannot be time dependent. What does the trend line represent exactly? Just the fact that when Lower Trop temps are higher, so are the Mid Trop temps? And vice versa? The trend line is at 45 degrees (or would be if the rectangles were squares), what does this indicate to you? What would it indicate if the trend line was at 60 degree slope? Or 30 degree slope? What conclusion do you draw from the plot and why?

The plot is anomaly to anomaly on a monthly basis and, as you said, shows that the climate noise (sigma?) that is affecting the signals affects the TLT and TMT and TTS and surface, for that matter, nearly the same.

A deviation from a 45 degree line would indicate different trends for the temperature anomalies being plotted. The UAH and RSS trends for TLT, TMT and TTS are different, but not necessarily by a lot.

beaker

Sorry Kenneth, I am genuinely trying to communicate my point, but perhaps being hampered by being too much the statistician and not enough a climatologist to have the right language.

The problem with the test is as much to do with the data as with the models. The Earth’s climate is the result of a chaotic system, with external forcings and internal feedback mechanisms. However, there is a chaotic element to the climate that is unpredictable, that depends on the initial conditions. This is why we can’t predict the weather more than a week or so ahead, because to make accurate one extra unit of time into the future, we (apparently) need to make exponentially more accurate measurements of the current atmospheric conditions. I understand there are longer term aspects of this chaotic behaviour, e.g. ENSO, that could affect the 20-year tropical trends that we are discussing. Even a perfect model can’t predict this component, so there is no reason to expect the model to predict the observed trend exactly (which is what is required to pass the +-2SE in the assymtotic case of an infinite, i.e. perfect, ensemble)

By “forced climate”, the IPCC seem to mean the average climate after you have averaged out the chaotic component. You can do that with the models by having an
ensemble, but sadly you can’t do the same with the observed climate.

Do keep asking questions, and I’ll keep trying to answer them the best I can, I may be near unintelligible, but I am determined! ;o)

MBK

Re: the various references to chaos. In deterministic chaos (which is itself a mathematical abstraction because the real world may add non deterministic noise) you will likely not be able to predict the trajectory / time-series of the system with any reliability except for the very short term. But, importantly, as long as the system remains in its previous basins of attraction, the attractors themselves will not change. So while climate models should be expected not to predict actual time-series, they should indeed be able to predict changes in the attractors: as a result of changes in model parameters (“external forcing”), does the model produce features of which measured real features are a plausible subset? And such features should include the kind of temperature differentials discussed here. Of course, “define plausible” is at the heart of this discussion.

To me this means it doesn’t matter in which exact initial condition some model is initialized. The initial condition only modifies the trajectory. But the attractor remains the same (for a given “external forcing”). I also don’t see why one would have to use an ensemble rather than one and only one “correct” model, with as many runs as necessary to get a good idea of the likely attractors (system features).

beaker

MBK #285: The fact that there is considerable spread in multiple runs for the same model (IIRC from the realclimate article), suggest the time span for the trend is short enough for things like ENSO to have an effect on the trend. More than that I can’t say.

If there more more than one attractor wouldn’t that make the test even less reasonable as the models can’t predict which one the real climate will have fallen into?

Kenneth Fritsch

Beaker, unless you can address this issue with more specifics of the climatology involved like the climate sigma and how it survives not being cancelled out by the use of ratios of trends, I do not think I will learn much from our discussion going forward.

Obviously we could make an educated guess or at least decide a reasonable measure of the climate sigma if one were to conclude that it does not cancel out by ratioing trends (as I think it does). Then instead of looking at X1-X2 with SEMs we would look at X1-X2 +/- CS. If CS is large compared to the forcings or mysterious in magnitude as it appears where you want to leave it, it would seem that the consensus of climate scientists is in much doubt.

Your argument as a statistician is that given a stochastic residual (or climate sigma) remains in Xobserved and is averaged out of the Xgcm then Xgcm – Xobserved can never equal zero. But as a statistician someone in climate science has to provide you with this given and also show that the residual is sufficiently large to make a practical difference in Xgcm – Xobserved failing all tests using SEM.

If we compared one group of climate model results with another group using large samples of both groups, we could take averages and use SEMs to determine whether the difference in means could happen by chance. We have effectively average out the climate signals in both groups. This appears to be a valid test that does not involve a residual climate signal. Now what are the chances that Xgroup1 – Xgroup2 = 0? If it isn’t zero does that mean the comparison cannot be made by this method?

I will conclude, by saying that you doing more than looking at the Douglas method strictly as a statistician, but instead are making assumptions about the climate science that you have not discussed or referenced in detail. At this point I have to go with Douglas et al. and assume that they can show that the climate signal cancels in their ratioing of trends or at least becomes practically small — or else why do the testing as they prescribe. I will patiently await an explantion from them or do more detailed analyses of my own to answer the questions.

beaker

Kenneth, the “sigma of the climate given the forcing” is NOT part of the models, it is uncertainty in estimating the actual forced climate (which we cannot observe) from the actual climate (which we can). It is part of the error bars on the data, not the models and ensembling can do nothing about it. This seems the key issue that I am not getting accross, if you get that point, the rest will follow.

MBK

beaker #287: Well from my understanding of chaotic systems a really good model would be rather like a risk analysis of how likely it is that the real climate will fall into a different attractor. Classic visualization, a ball in rough terrain. It will fall into a depression (the attractor) and if you move it most of the time it will fall back into the same “valley”. But if you move it too much, it may roll into a neighboring valley. It seems to me though that climate models are far from any such predictive capability. The very idea that there is a trend to be predicted (albeit with noise overlaid) is somehow quite incompatible with the idea of a system exhibiting chaotic nonlinear dynamics, possibly over a large range of scales (self similarity). Such a system would make any effort at predictive modelling look naive.

Somehow I doubt though that within the time and parameter scales we’re interested in the system is really all that chaotic. It looks rather homeostatic to me, it is rather its relative stability that is surprising. If this is true then models do have a chance, and a point. So this leaves the door open to models or observations that predict a departure from a previous homeostasis within its random-noise bounds. On the other hand this also means then that models can’t take the chaos excuse anymore. They do have to predict something useful and testable.

For the case in point here we’d ideally want to know what the “random-noise bounds” of the temperature differential were, w/o GHG additions, over a time-scale much longer than the one we are trying to test. Just common sense – yes we’d still only have one real climate “run” but over a period spanning many of the lengths of time we’d like to know about, so we can assess typical variability over any one length of time. Sadly, we don’t have much data here – not over time scales of historical, plausible variability. But it is precisely this that models should actually be doing in the first place – to put expected bounds on natural variability, and contrast them with expected departures from this variability, complete with significance levels and predictive power (what is the smallest departure from normality that can be asserted with good confidence within a certain time frame?). And this kind of data precisely is what multiple model runs give, so multiple model run data analysis is crucial IMO.

What I don’t understand is why these data do not seem to be discussed, why models aren’t understood as research tools rather than as forecasts. The first thing a model should do is to give an assessment of expected variance in the output (one model, many runs). Without that, you can’t determine predictive power. The model output becomes untestable.

Jon

@286

Obfuscation appears to be your forte.

Are you kidding?! He is going out of his way to answer and re-answer (and re-re-answer) the questions posed to him and avoid falling into a false dilemma when a yes or no is insufficient. How can that possibly be called obfuscation?

His patience is remarkable.

He also appears to be quite capable of discriminating between ensemble projections and forecasts, which is more than can be said for quite a few people around the blogosphere.

Ron Cram

beaker,
re: 290

You wrote:

Douglass statement you give is vague and informal, I agree with it to an extent, only a few of the models give a reasonable approximation to the data, but without knowing all of the uncertainties, especially the “sigma of the climate given the forcing”, I wouldn’t go as far as “reconcile” as that would imply more than the plot can support (as there are no error bars).

The statement by Douglass we are discussing is not “vague and informal.” It is a challenge:

Invent your own tests or criteria. The main conclusion will be the same – only a few models can be reconciled with the observations.

The “sigma of the climate given the forcing” is irrelevant to this discussion. We are not talking about climate sensitivity. The issue is “Are the models and the theory consistent with observations?” The models and the theory call for faster warming of the tropical troposphere than at the tropical surface.

From your answer, I can only assume you are unwilling to attempt any mathematical effort to salvage more than the three models Douglass left standing. Is that correct?

Sam Urbinto

I think I figured this out.

beaker, your #292 I never said you cared if they pass or fail and I don’t care either. 🙂 And you’re right, what test do you perform? The question is if test A can reconcile models and observations, isn’t it?

Pick test A.

If this paper is trying to show that models and observations can’t be reconciled using SEM, does it do that or not? Not that it’s trying to reconcile them, just to see if it can be done using SEM.

I can pick any test and number of models and see if I can reconcile. Could I pick SD-SD? Sure. But no, too easy to pass. Could I pick SD-SE (or SE-SD)? No, I have to guess at variables. Could I pick individual observations? No, I already know if it passes or fails depends on the individual test and which I pick (which most of the time would be fail).

So question; does this test show that if you pick this group of 22 models you can’t use SEM to compare to observations? Does it falsify the usefullness of SEM to combare this ensamble to those observations?

Kenneth Fritsch

Beaker, this “sigma of the climate” that you have latched onto from a Steve M expression used recently at CA appears to be taking on a growing view from you of an unknown and unknowable quantity with a description so vague as to take on multiple meanings.

Please give me a good reference to its description and definition and how it would manifest itself differently in the models than in the observed climate using ratios as in the Douglas paper and, if possible, a range of the magnitude of its effects.

I thought you were differentiating the model ensemble output from the observed as the ensemble being merely the forced residual after averaging out a sigma of the climate, but I understand you are now saying that the models do not include a sigma of the climate but the sigma of the climate in the observed is related to the forcings (and initial conditions?).

Taken in total I think you are saying that all the models can do is give a potential measure of the forcing but only to an approximation because only in observed case does the unknowable sigma of the climate manifest itself — and from the forcings.

The unknowable sigma of the climate in a unique manifestation of an observed climate run (limited, of course, to the single run we have and will experience) would then appear to make a statistical comparison of a model output, much less an ensemble, with the observed, a worthless, or at best, a very uncertain proposition.

Jon

I’d like to learn this discrimination; in the former the goal is to find predictive distribution of temperature given the forcings and in the latter just marginal distribution of future temperatures ?

Read AR4 WG1 Chapter 10 and then look at a paper like Keenlyside 2008. The former is an attempt to show the longterm warming trend by smoothing internal variability the latter tries to anticipate. In the Keenlyside example, they use sea surface temperatures to attempt to model variability of the MOC. Such variability in an unforced climate would look like noise- fluctuations up and down that over time had a flat slope. Increased greenhouse forcing doesn’t eliminate that variability- you still see the fluctuations. However it gives the overall picture a positive slope. Ensemble projections don’t try to predict the fluctuations, they show something more like the slope.

A simple visual example would be comparing the variability of PDO to temperature. The PDO fluctuates, as does temperature, but temperature does so with a positive slope whereas PDO as an oscillation has no long term trend.

If we look at an ensemble projection you can see even multiple runs of models will result in fluctuations, but averaged across all models the mean smooths them out, showing the overall trend.

Sam Urbinto

So again, does this test show that if you pick this group of 22 models you can’t use SEM to compare to observations? Or doesn’t it?

Willis Eschenbach

beaker, thanks for hanging in. However, you have consistently refused to answer my question. I’m a patient man, however, so I’ll ask it again:

The modelers agree that their models can’t predict an “individual realization” of the climate system. However, they say that they can provide a model of the long term climate trajectory. Here’s a typical statement of the situation, from Professor Alan Thorpe, the head of the National Environment Research Council. Dr. Thorpe is a world renowned expert in computer modeling of climate. He says (emphasis mine):

On both empirical and theoretical grounds it is thought that skilful weather forecasts are possible perhaps up to about 14 days ahead. At first sight the prospect for climate prediction, which aims to predict the average weather over timescales of hundreds of years into the future, if not more does not look good!

However the key is that climate predictions only require the average and statistics of the weather states to be described correctly and not their particular sequencing. It turns out that the way the average weather can be constrained on regional-to-global scales is to use a climate model that has at its core a weather forecast model. This is because climate is constrained by factors such as the incoming solar radiation, the atmospheric composition and the reflective and other properties of the atmosphere and the underlying surface. Some of these factors are external whilst others are determined by the climate itself and also by human activities. But the overall radiative budget is a powerful constraint on the climate possibilities. So whilst a climate forecast model could fail to describe the detailed sequence of the weather in any place, its climate is constrained by these factors if they are accurately represented in the model.

Thorpe, Alan J. “Climate Change Prediction — A challenging scientific problem”, Institute for Physics, 76 Portland Place London W1B 1NT

If individual realizations are not predictable, and “the key is that climate predictions only require the average and statistics of the weather states to be described correctly”, then we must ask whether the increase of temperature trends with altitude (which is both predicted by theory and “confirmed” by models) is an “individual realization”, or whether it is an “average or a statistic of the weather.”

Since we are looking at averages of what has happened over a relatively long period (20 – 30 years), it seems quite clear that it is an “average or a statistic of the weather”. This point of view is strongly supported by the fact that the increase is predicted by theory – in other words, it is not the result of a given “individual realization” of weather. It is something which is determined by the physics of the situation, not just in a general sense, but to the point where we can make theoretical predictions, based on the AGW hypothesis, of both its sign and its value.

Since it is an “average or statistic of the weather”, the standard error of the mean is the appropriate metric to use for comparisons. The idea that one model out of twenty-two getting this critical parameter kinda sorta right somehow vindicates the models is contrary to everything we know about averages. The individual model results are long-term averages of individual runs of each model. The averages of the model results are averages of long-term averages. The averages of the observational data are also averages of long-term averages. None of this is an “individual realization”, they are all “averages and statistics”.

I also note in my re-reading that two other people asked you to answer this question …

All the best,

w.

PS – beaker, you say:

The models have NOT been falsified, they have been shown to be significantly biased, I have explained the difference (and at Willis’ request demonstrated that these terms have pre-existing meanings in statistics).

My friend, you are trying to rewrite history. First, the term you used before was not “falsified”, it was “inconsistent”, so your claim that I requested you to demonstrate anything about the term “falsified” is … well … not to put too fine a point on it, your claim is falsified.

Second, in response to my request, you admitted that the “pre-existing meaning” of “inconsistent” was NOT the meaning you were using. Despite that, you have continued to use it as though you were vindicated, and have continued to make your incorrect claim that you were using “inconsistent” in some standard statistical manner.

I hardly think that either of those statements supports your case. I was impressed, however, by your clever use of words. You say you “demonstrated that these terms have pre-existing meanings in statistics”, which is 100% true, you did demonstrate that … but somehow you forgot to mention that you were NOT using the pre-existing meaning, you were using another meaning altogether …

Jon

you didn’t answer the question. In fact, you managed to write your entire post without mentioning “forecast” even once.

I cited Keenlyside 2008 as an example of forecasting up the thread (240), I didn’t think I needed to repeat myself when referring back to it.

So, once again … what is the difference between an “ensemble projection” and a “forecast”?

The ensemble projections are taking in some cases multiple in other single, depends on the model, runs and averaging them out in an effort to reduce internal variability. A forecast unless otherwise stated is an attempt to actually capture a reasonable amount of variability. Interestingly, although Hansen’s 1988 projections were not intended to be forecasts, as single realizations rather than ensemble means they functioned similarly- and in them you can see quite a bit of variability compared to the way projections are handled now (compare them to Figure 10.5 in the AR4).

Anyone who cites a whole entire chapter of the IPCC Report is no better than a Bible-bashing evangelist who screams “The answer’s in the Book, in Leviticus”. Well, maybe it is, but where?

It’s on page 762, in discussing ensemble projections smoothing out internal variability.

Look at that section, look at the Keenlyside paper. The kind of variability that they describing as dampening the warming signal is smoothed out in the projections given in Figure 10.5. A forecast would have the kind of fluctuation we see in actual temperature whereas the mean ensemble projections would be closer to the slope of the temperature increase.

The first quote you cite would refer to a paper like Keenlyside using an ensemble of multiple models run with the same or at least similar inputs in initialization for sea surface temps in order to best depict this initial state’s impact as variability. Not an ensemble mean for projections run under differing initializations in order to smooth out variability. I assume the second is talking about the same- that mean ensembles under identical initializations should give a more accurate forecast than a single model because errors in the model would be averaged out.

I apologize if this doesn’t seem clear enough- it’s hard for me to describe a more clear way of distinguishing between the two than comparing the description of the projections to a paper like Keenlyside’s.

multi-model “ensemble projections”

Would be the forced component, with internal variability and model errors basically averaged out.

and “forecasts”

Using specific initializations to attempt to capture internal variability.

multi-model “ensemble forecasts”

Multiple models using the same initializations so that internal variability is present but modeling errors are averaged out.

finish up with the famous “single model ensemble forecast”.

I don’t understand this.

beaker

Good morning all!

Patrick M #296: As I see it, the most useful thing the models tell us is the consequences of the modellers assumptions, and more importantly, the spread of the ensemble gives an indication of the uncertainty both within the modelling community (the systematic uncertainty) and the uncertainty that arises through the simulations themselves (the stochastic uncertainty). As a result, I am happy to see money spent performing as many simulations as possible on as many models, to get as good a representation of the uncertainties involved as possible.

Basically, I would not recommend deleting models as this makes the ensemble appear to give an unduly confident impression of the consequences of the modellers assumptions, which I think is in itself dangerous. It seems to me to be a form of cherry picking. These are not statistical models and should not be treated as if they are. When looking at a set of “forecasts”, we have to remember that this “forecast” is just a statement of the logical consequences of a set of beliefs about the way the climate works, nothing more. Whether you believe the forecasts should be based on whether you accept the assumptions rather than because you believe the “forecast” itself. Whether the “forecast” is a useful predictor of future climate (under a particular scenario) depends on how close these assumptions are to the reality (but I can’t comment on that, I don’t have the expertise).

Now if the modelling community decide that the assumptions underpinning a particular model are unreasonable (perhaps as a result of reading Douglass et al. or the RealClimate article), then that would be a good reason for deleting it from the ensemble.

Ron Cram #298: It seems to me that we are using “sigma of the climate given the forcing” to mean different things. I am using it (as I believe Steve was, but perhaps I am wrong) to refer to the uncertainty in inferring the forced change of the real climate (which we can’t observe directly) from the actual climate observations.

If the issue is “Are the models and the theory consistent with observations?” then I would say that the RealClimate +-2SD test is the correct test of this question. That is because the theory has uncertainties involved, which are represented by the spread of the ensemble. If the spread of the ensemble overlaps the spread of the data, then there is “no clear inconsistency”, but the wide error bars tell you that the theory is very uncertain. This isn’t “playing with words”, in science (and statistics in particular) we have to be very careful to set out the question completely unambiguously so that we can interpret the answer correctly.

All of the ensemble is useful in telling you the consequences of the modellers assumptions. That is the real purpose of the ensemble. That the ensmeble only just manages to explain the observations casts doubt on those assumptions, but doesn’t falsify them outright. As I have said before, I am not keen to delete any model from the ensemble as this invalidates the most useful intepretation of the ensemble projections.

Michael Smith #300: If you had a perfect model with unlimited resolution it still could not be expected to predict the observed climate exactly as the observed climate has an unpredictable chaotic component. The best it can do is to run the model lots of times to get an ensemble of predictions that are all consistent with the physics, but varied due to differences in the initial conditions of the simulation. Now if you had the perfect model and ran an infinite number of simulations, you would be able to find one of them that was the essentially identical to the observed climate. That shows that an infinte ensemble using the perfect model can be reconciled with the observations as the observations lie within the uncertainty of the model.

This model will fail the +-2SE test because the 1/sqrt(2) factor will mean that the SE is zero. The fact that a model that has been shown to be consistent fails the +-2SE test shows the +-2SE test is incorrect, not the model.

The SE is inappropriate as it doesn’t actually measure the uncertainty of the ensemble prediction, but the SD does. Therefore if you ask if the observations can be reconciled with the models (in statistical terms is there an overlap in the uncertainties of the models and the data) then you need the SD.

Kenneth Fritsch #301: I have given a definition of what I take “sigma of the climate given the forcing” to mean (in my comment to #298), I can’t see how it can be measured, if you can suggest something, I am happy to discuss it with you. It is an uncertainty in inferring the effects of the forcing on the real climate without the unpredictable chaotic component.

As I keep saying, the “sigma of the climate given the forcing” does not manifest itself in the models, it is a property of the observations, not the models.

The perfect model (with unlimited resolution) can predict the actions of the forcing on the climate with complete accuracy, but it can’t explain the chaotic component which depends on the initial conditions. That means it can’t predict the observations exactly, because the observations are not just the forced component. That is why it can’t pass the 2SE test (even though no model can do better).
gp #306: That would be a valid test, but it is only a test of the ensemble mean, not a test of the ensemble itself, which is an important distinction.

I am not making any estimation of the size of the “sigma of the climate given the forcing” as I am not a climatologist, however as a statistician I can see that it only needs to be non-zero for the perfect model to fail the +-2SE test.

As it happens the measurement uncertainty puts fairly large error bars on the observations already, just look at the disagreements between the observed trend computed from different sources.

Your slicing method might give a useful ball park estimate, however, as I said it only needs to be non-zero for the +-2SE test to assyptotically fail a model it should pass (invalidating the test).

Willis #307: Averaging (whether over time or over realisations) can only attenuate the effects of the chaotic component of the individual realisations (including reality), not eliminate it altogether. This means there will always be some residual uncertainty in estimating the forced climate. All that is needed for the perfect model counter-example to invalidate the 2SE test is that this uncertainty is non-zero, which arguably it is.

I have seen arguments that show that ENSO can a significant effect on ten year trends (1998-current being the most frequently cited example), I can’t see a good argument why it should have no effect whatsoever on a 20 year trend.

I have noted that this question has been asked more than once, and I have tried to answer it more than once, even if it doesn’t appear that way!
PS, for me showing that the models are inconsistent falsifies them, I was trying to make that distinction in what the Douglass inconsistency test actually shows.

I pointed out that “inconsistent” has a meaning as a term, it is the phrase “inconsistent with the models” that is the problem. It is quite possible that I have not explained this very well, but to a statistician to say “the data are inconsistent with the models” has a particular severity, much more than “the models do not provide a good fit to the data”. Douglass et al. certainly demonstrate the latter, but not the former. Does that make it any clearer?

Willis #310: While I might apparently appreciate the difference between a projection and a forecast, I am not sure that I understand the nuance in those terms as used by climatologists. I was using the “” to hint that I was not entirely happy with the terms as I understood them, but was trying to give as helpful an answer as I could.

I think I have made several statements about how I think the ensemble should be used (and hinted that an audit of IPCC recommended practica maybe in order). The average of the runs does not capture the variability, but the SD error bars would to an extent. It could well be that the IPCC are not making full use of the models.

regarding the multi-model ensemble versus single model ensemble, the reason for ensembling over models is just the same as ensembling over runs, it averages out (or if you are a Bayesian captures) uncertainties. In the former, it is the systematic uncertainties and stochastic uncertainties in the latter. The same benefits accrue in both cases. On average the mean will be a better predictor, but not neccessarily better than the best of the components.

Sam Urbinto #299: You chose the test according to the question you want to ask rather than the answer you want to get. As I have pointed out, you can try and reconcile the models and data using SEM, but it is pointless as the best model that you could possibly make is almost guaranteed to fail it, which shows that the test can’t give a sensible answer to that particular question.

SEM is the wrong statistic for this question. The perfect model example demonstrates this as the test gives an obviously incorrect answer to the question. The reason why it gives the wrong answer is that it treats the SEM as the uncertainty of the ensemble, it isn’t, it is only the uncertainty of estimating the true mean of some putative underlying distribtion of models given this particular sample of 22 models.

MrPete

beaker, thanks for your ongoing interaction!

A couple of comments:

1) Seems to me that if there’s a significant chaotic component to climate, we’re in trouble because natural variability can be huge. And AFAIK none of the GCM’s attempt to model a chaotic component.

2) See my post above — GCM’s are not (yet) validated as approximations of reality, so there is no reason to presume that an infinity of runs would incorporate the correct future. Important systems are not modeled (by anyone), or badly modeled (by everyone), or there’s just plain bugs (somewhere–and without good software practices, nobody really knows where but experience says the bugs are there to be found).

All this to suggest the CI of the GCM’s is quite large and not derivable from any attribute of model performance.

For example, the ensemble results says a little about modelers’ assumptions. What do
historic climate excursions really look like? We don’t actually know. A 30,000 foot view says: climate trends are said to require a few decades to show up; we only have thermometry and satellite data over one 30 year period and even that data is suspect; temp history over 150 years has proven to be quite suspect (for how many years has the global 1900-1950 temp record been a stable and non-controversial data base? Zero. Compare that to what we know of other observable phenomena.)

Thus (big picture here) don’t we have little with which to confidently validate GCMs, statistically or otherwise? Aren’t there too many unknowns?

Geoff Sherrington

There is an even better estimator than the Willis Eschenbach # 308 “single model ensemble forecast.” Maybe it is the “ultimate precision single model ensemble forecast” as shown by modeller 15 in the paper of Douglass et al, thus:

Model 15, first column. Average of 22 mosels next, then SD next column.
163 156 64 (SURFACE)
213 198 443
174 166 72
181 177 70
199 191 82
204 203 96
226 227 109
271 272 131
307 314 148
299 320 149
255 307 154
166 268 160
53 78 124 (100 KpA)

I know I have posted these figures before, but here is a question: Can the mathematical model be TOO good?

I think that model 15 comes very close to the mean of all the other models, but I do not think one can draw significant inferences about whether it represents Nature.

It surprises me that pro-model advocates have not triumphed this achievement as a vindication of claimed modelling prowess. One part in a thousandth of a degree per decade, agreement at 3 different altitudes? Who needs multiple runs to reduce error?

Michael Smith

Willis, 314 was a wonderful summary. Thanks!

RomanM

#311 beaker

You have consistently repeated a number of statements which some of us have trouble accepting. In my view, part of the reason is that there are implicit assumptions about the situation we are discussing that are needed to justify those statements. If you have studeied statistics, then you understand the need in an analysis to specify what assumptions you are making before you follow a particular statistical path for that analyis. Whether or not the analysis is reasonable depends on how realistic those assumptions are. I have selected several quotes (not necessarily in the order made and bold added to parts to emphasize what needs assumptions for justification) from your recent post which I feel could use some explanation of the assumptions you made in stating them.

regarding the multi-model ensemble versus single model ensemble, the reason for ensembling over models is just the same as ensembling over runs, it averages out (or if you are a Bayesian captures) uncertainties. In the former, it is the systematic uncertainties and stochastic uncertainties in the latter. The same benefits accrue in both cases. On average the mean will be a better predictor, but not neccessarily better than the best of the components….

All of the ensemble is useful in telling you the consequences of the modellers assumptions. That is the real purpose of the ensemble. That the ensmeble only just manages to explain the observations casts doubt on those assumptions, but doesn’t falsify them outright. As I have said before, I am not keen to delete any model from the ensemble as this invalidates the most useful intepretation of the ensemble projections….

Basically, I would not recommend deleting models as this makes the ensemble appear to give an unduly confident impression of the consequences of the modellers assumptions, which I think is in itself dangerous. It seems to me to be a form of cherry picking. These are not statistical models and should not be treated as if they are. …

What assumptions are you making about the ensemble of models? Are they a random sample of models from a population with some underlying distribution or are they just a fixed finite set of models? Why is the average a meaningful quantity? What stochastic uncertinties are averaged out and why does this produce a better predictor (presumably of the climate or any of its parameters). What in the underlying structure relates the ensemble in a numeric way to it to the climate and/or its parameters? Why is the SD of the ensemble meaningful? What does it estimate (if anything)? What specifically makes +/- 2 SD meaningful? Why does deleting models which are obviously inept invalidate the most useful interpretation of the ensemble projections (and just exactly what is that interpretation)? If these models do try to predict or project a “chaotic” climate, why are they not “statistical” (or is there an EKT definition from the statistics community involved here as well)?

A general observation on the Douglas paper: What they are evaluating is the ability of the models to estimate a set of climate parameters, the trends of the temperatures. No one is asking that the models “project” or “predict” the daily or monthly temperatures or duplicate the climate through an infinite set of runs in a monkeys and typewriters writing Shakespeare sort of way. If they can’t, then there is a problem. Do you think that that particular aim was achieved?

Patrick M.

re 311 (beaker):

You state:

As I see it, the most useful thing the models tell us is the consequences of the modellers assumptions, and more importantly, the spread of the ensemble gives an indication of the uncertainty both within the modelling community (the systematic uncertainty) and the uncertainty that arises through the simulations themselves (the stochastic uncertainty). As a result, I am happy to see money spent performing as many simulations as possible on as many models, to get as good a representation of the uncertainties involved as possible.

Regarding the bold text: and then what? Would you at some point consider thinning the field of GMC’s based on performance?

beaker

Good evening all (one last post for the day)!

Roman M #321: That is a lot of questions, the reason why an ensemble is a good thing is a bit involved and maybe requires a new thread, I’ll give it some thought and get back to you, but in the mean time…

As to your general observation, yes indeed the general aim was achieved, I only have problems with the (statistical) methodology and the lack of a distinction between “the models” and the “mean of the ensemble” in many places in the paper.

Ron Cram #323: I have pointed out that I don’t think the ensemble should be whittled down at all, that means that both “yes” and “no” are incorrect answers. Your question is a logical fallacy as it assumes that the ensemble ought to be split up, which is not a premise with which we both agree. See http://en.wikipedia.org/wiki/Mu_(negative) for further details.

As to whether that view is “outrageous”, perhaps you should consider why I might think that the very wide 2SD error bars of the models highlights their defficiencies more clearly than the test in Douglass et al. If you understood that point, you would understand why you wouldn’t want do anything to make them narrower.

Patrick M #324: The only way that the ensemble should be thinned is as a result of a change of opinion on the paert of the modellers. Say a modeller reads Douglas et al and thinks “he has a point there (although the stats are wrong ;o)”, and then goes back to the drawing board to work out why the models don’t do what the theory and observations suggest they should. He then changes his opinions on the physics of the climate and this is reflected in the next (hopefully more accurate) generation of models. It seems to me that the safest way to treat the models is as a statement of what is plausible given the modellers best understanding of the physics (which will only be accurate if that understanding is good). The huge error bars basically mean that we have to consider a wide range of possible outcomes and that appropriately guards against “jumping to conclusions”.

N.B. My (rather cautious) view on how the models should be viewed is separate from the discussion of whether the 2SE test is valid, neither is contingent on the other.

Patrick M.

re 326 (beaker):

Looks like my post 327 might be along the lines of your answer in 326.

Sam Urbinto

beaker #311 “You chose the test according to the question you want to ask rather than the answer you want to get.”

And the answer I want to get is “If I assume these 22 are perfect models, can I use SEM to reconcile them to actual observations?”

And as Douglass et al and you and others have shown, the answer is no. That the models do or don’t agree with each other, or if the SD shows they’re uncertain or not is beside the point.

Inappropriate, meaningless, uncertain and conflicting.
(SEM, SD, SEM/SD, by model)

If my question is “Can I use SD to show the models have very wide 2SD error bars” or “Will SD show the models are defficent?” are different questions.

But I do agree (#326) that the models shouldn’t be broken up, take them all and see what “the models” tell us.
Which is you can’t compare them with SEM and get reconcilliation, and SD shows them defficient.

Why do you have such an issue with why the paper didn’t try to answer the question leading to the second, I have no idea.

MrPete #313 “Thus (big picture here) don’t we have little with which to confidently validate GCMs, statistically or otherwise? Aren’t there too many unknowns?”

Yes. I would say so.

Willis #314 “But the observations say it’s not warming aloft, it’s cooling compared with the surface. Not only that, but the higher you go, the greater the cooling.”

Then the models showing warming are wrong. No statistics needed. As far as why, well, how much air are we moving, and how saturated is it, and what temperature is it, and what is the pressure at the location? 🙂

http://goldbook.iupac.org/A00144.html http://raob.com/pix/EmagramB.JPG

Willis #314 “The point is that observations of negative and decreasing trends aloft is a direct refutation of the theoretical basis of the claim that the warming is caused by GHGs.”

“Extra warming” (aka rise in the anomaly trend) 😀

Ralph #317 “Could someone correct the errors in my reasoning here?”

I’d say with model comparing papers and tests and the like, what gets done depends on the assumptions made. Even if other people disagree with those. Or not. I think you basically have it correct as to what was done and what was assumed/decided upon/whatever.

Ken #329 “..that looking at the ratios of trends, as in the Douglas case, mitigates the objection to the chaotic component in the observed climate.”

I would think it would. I’d be interested in a short answer as to why that’s an unreasonable way to go about things.

But BTW, it’s bender who’s a POV! 😀

Sam Urbinto

There’s an interesting tidbit here in this paper Phil linked elsewhere.

Arctic sea ice in IPCC climate scenarios in view of the 2007 record low sea ice event
A comment by Ralf Döscher, Michael Karcher and Frank Kauker

It is necessary to be aware of an important difference between the observed data and climate model results as they typically are published. Mostly, as is the case in Fig. 1, the presented climate model
results are ensemble means. In this context an ensemble is a number of model experiments which
differ from each other, e.g. by changed initial (starting) conditions and in the case of AR4 by
different model formulations.In coupled climate models small initial differences may grow large as a consequence of the non-linearity and chaotic behavior of the climate system (the famous butterfly effect) and due to different model sensitivities. The use of ensembles instead of just one experiment provides a range of possible realizations of the climate system under given external forcing. It is evident that the ensemble mean provides a more robust picture on the system’s long term response to the external forcing than any single ensemble member. On the other hand the ensemble mean has a reduced variability and less extreme events than any single member. For a test whether the IPCC models are ‘too conservative’ in comparison to observations, we should compare the observed ‘realization’ with single realizations of the ensemble models.

steven mosher

re 331. ya UC I raised that issue before but nobody seems to get it.

One set of rules for attribution studies and another set for rejecting models.

steven mosher

RE 336.

I found the GISS hindcasts for 1880 to 2003 ( actually gavin pointed me to them)
You can pull out various forcings and see the reponse. Today I thought It might
be fun to compare the 4, 30 year periods. 1880-1910, 1911-1941, 1942-1972, and
1972 to 2003. And look at the metric of GSMT trend, subjectively, ModelE missed
on the first two climate periods, not even close, Nailed the flat trend from 1942
to 1972, and nailed the rise from 72 to 2002. so, 2 otta 4 aint bad.

Sam Urbinto

mosh #334

But the SRES is a guess scenario-driven “storyline” and even then, if they’re fro emission inputs, that’s just more GHG hooey factors, right?

What about the heat and lower albedo involved with those two and “urbanization” (and everything that comes from it animals, farms, cities, freeways, suburbs, etc)?

A case of “It must be GHG because we know they absorb IR.” it seems to me. That it’s a fairly well known static thing, therefore “it must be it” in the system. Not that they can’t explain it any other way, just that it’s too difficult to deal with?

But, hey.

Kenneth Fritsch

Re: #336

Steven Mosher, could you share that link with us or do I have to go back to web site where I noted that Gavin graciously gave it to you after a reminder. 2 out of 4 heads on coin flip is pretty average, so I have to see these 4 periods’ fits for myself. Trust but verify.

Geoff Sherrington

Re 333 Steven Mosher

Do you infer that by nailing 2 of the 4 periods in the hindcast, the modellers knew which physics/chemistry caused upturns and downturns in a quantitative manner? If this is so, then it is the first admission of this that I can recall.

steven mosher

re 337

http://data.giss.nasa.gov/modelE/transient/climsim.html

table 1, select all forcings, select lat-time as the response

next page select a running mean of 1 month, select output options that give
you a table, you’ll get mothly data from 1880 on. i think i started with 1881
I then got hadcru data, make sure you adjust to the right baseline ( hadcru is
61-90

Then I just did simple linears on the first 30, next 30, next 30, last 30
please double check, i did it quick and dirty

steven mosher

re 335. not sure I follow sam. the sres are storylines about future emmissions. the
hindcast from 1880-2003 uses “historical” forcings.

Ron Cram

beaker,

Regarding (iii) (iv) and (v), I agree with Michael in #340. The issue is not whether the difference in the observations is less or more than 1e-6 K/decade. It does not have to do with any individual measurement. The issue is whether the troposphere is warming faster than the surface or not? This is a meaningful difference and is a sufficient basis to reject models that fail the test.

I have to reject (vi) as well. While the statement by itself is true, it does not apply in the present case. Someone may well argue that weather is stochastic and monthly averaged temp differences between troposphere and surface may not be uniform, but the process is certainly not stochastic over longer time periods. It is deterministic. The idea the troposphere will warm faster than the surface especially in the tropics is central to the current theory of AGW. We have more than 20 years of data to compare to the models. It is obvious the vast majority of the models are consistent with the theory but not with observations.

steven mosher

re 344, also note that I did a fairly easy test to hit. a 30 year trend, some features
like the early 20th century warming were not picked up by the model. this could be due
to errors in the historical forcings.

RomanM

#338 beaker
Let’s examine your statements one by one:

(i) The SE test is a test of whether there is a statistically detectable difference between the observations and the ensemble mean.

No, the test is whether there is a detectable difference between one or more parameters of the population of temperatures and the ensemble mean. More specifically, the test decides whether a suitably selected sample from a population of models whose mean is actually equal to the values of the parameters could reasonably generate the calculated ensemble statistics or whether the difference is extreme enough that this would be unlikely to occur.

(ii) It would be unreasonable to say that a model could not be reconciled with the observations if the difference between the ensemble means and the observations was less than 1e-6 K/decade.

True or not, irrelevant to the issues here.

(iii) An ensemble with finite variance will fail the SE test if it is sufficiently large, even is the difference between the ensemble mean and observations is less than 1e-6 k/decade.

Nonsense. This same criticism of statistical testing could apply to every statistical test. All you are saying is that given an appropriate set of assumptions, if you have enough information, you can detect a difference as small as desired. It doesn’t mean that for large samples only small differences can be seen. Of course, according to the data in the paper, we are talking an ensemble of about 1600000000 models to detect your chosen difference.

(iv) Therefore the SE test doesn’t tell us whether there is a meaningfull difference between the observations and the ensemble mean

I repeat, it gives us insight into whether the observed differences could be accounted for randomness or whether there is a real systematic difference. “Meaningful” requires an external criterion to be applied to determine what difference you are willing to accept. Confidence intervals and other methods can be used to decide the size of the difference.

(v) Therefore the SE test doesn’t show if the ensemble mean can be reconciled to the data in a meaningful way.

I guess that means that every test involving population means (not just in climate science) fails in exactly the same way, if you take too large a sample, and decide that there is a difference, then that difference is meaningless.

(vi) The use of an ensemble reduces the stochastic uncertainty of the models.
(vii) The larger the ensemble, the more the stochastic uncertainty is averaged out.
(viii) The more the ensemble averages out the stochastic uncertainty, the more likely it is to be rejected by the SE test.

I am still waiting for the assumptions that you are making about the ensemble so that these last statements have some validity.
Basically, this whole issue appears to me as a specious red herring.

IMHO, if you want to test individual models, you do that by comparing the characteristics and output of each model separately to the observed data. The ensemble can at best (under the right conditions) tell you how the various models differ among themselves, but offers little or no information about the individual models. Assuming all models are equally uncertain is pretty much baseless – the lack of existing individual information to make determinations about model validity (because none is provided by the modelers) is no excuse. If a given model is deterministic, i.e., then every run from a fixed set of initial conditions gives the same result, so at best it seems that you can look at the model’s behavior as the conditions change. That is the appropriate “ensemble” within which to evaluate that type of model. However, questions of applying inferential statistical methodology to the ensemble do arise. For models with stochastic components, the appropriate “ensemble” is multiple runs of only that model under each condition. Then, it makes sense to try to do proper statistical evaluation.

UC

The larger the ensemble, the more the stochastic uncertainty is averaged out.

More studies, more accurate results. http://www.dilbert.com/2008-05-08/

Kenneth Fritsch

Beaker @ #338:

putting to one side for the moment my views on how the ensemble should be interpreted and the assymptotics as that seems to have us getting bogged down in the details, rather than the key point, which is that the 2SE test is arbitrarily hard to pass.

The devil is in the details and that is where the real and practical world lies. You have made your general POV on the matter of using SE and the chaotic nature of the observed and models being sensitive to the initial conditions very clear.

However, you have not replied to the proposition that in the Douglas case where ratios of trends are being used that the chaotic content in the observed and models does not apply, i.e. it is essentially differenced out.

Your concentration on comparing means with large sample sizes and the means not being exactly equal makes it sound as though statistical tests taking sample size into account using SE measures are never or should never be used.

I have noted previously that an SEM test could apply to your idealized situation (not necessarily to the Douglas case because it may not need it, but the general case) of the the observed if one could estimate the chaotic residual (E) remaining in the observed that would be averaged out in the model mean. Then one has a comparison using SEM where the Xmodel – Xobserved = 0 is no longer expected but absolute value of Xmodel -Xobserved = E, where Xmodel is the average of the ensemble.

I think the discussion has reached point where it would be instructive to discuss details and references — and maybe with some insights into why modelers prefer evaluating their models the way they do.

Cliff Huston @ #346:

If the definition of the perfect model ensemble is that it exactly matches the real world, by definition it will pass the test whether you use 2SE or 2SD

Beaker’s point would be that the observed is more like another model output and the chaotic content or climate sigma of the observed will vary the output which in turn is averaged out to some degree in the ensemble mean. Therefore Xmodel – Xobserved = E.

As stated above I have proposed that that condition does not necessarily appy to the Douglas case and that one can test for differences in the general case by assuming that the difference could be as large as E.

I think on initial pondering of this issue that if one does not do some of this estimating of E, in the general case, that one would conclude that computer outputs could never be reasonably compared in a statistical sense to an observed value. One could make predictions/projections/whatevers of the future climate with model outputs and then point to the range of outputs with comments like they all show an increasing temperature and here is the range. Of course what is avoided here is any comparison to an observed. Lacking a rigorous statistical test I would suppose a proud modeler would say that that agreement and range is sufficient for policy makers to go do their work. When organizations such as the IPCC get involved they need to push the concept of uncertainty a step further and add in essence an show of hands by (some or the) experts.

I was hoping that Beaker would give us some insights into how the statistical testing of uncertainty of model outputs and comparisons to observed values is and/or should be progressing.

Kenneth Fritsch

Re: Steven Mosher @ #342

re 339. I infer nothing, first have kennth check my work in case I botched something
there were peroids when the hindcast was off, missing bumps and such. I havent spent much
time looking at it, you can select different
forcngs and see the reponse.

No, no Steven, that’s my line to get people to do my work — and that has not worked for me here.

Thanks for the link that your pal, Gavin, provided. I’ll look at it, but do not intend to preempt your posting on it first.

JamesG

The argument that ensembles are more accurate than individual models is of course the argument that employs the greatest number of climate modelers. No longer can your model be total cr*p if it is a vital cog in the wheel. An appeal to a greater number of models to improve accuracy is only an appeal for yet more money, when in fact they need to reject the really bad models, disband the modeling groups responsible and focus on just the semi-bad ones. Seems to me the Russians are the winners of this contest and I’m not surprised.

Sam Urbinto

How about something like this:

We have tested the proposition that greenhouse model simulations and trend observations of various altitudes of the tropical troposphere can be reconciled. Our conclusion is that comparing the standard error of the mean does not reconcile the two with each other. Other comparisons may yield different results.

Sam Urbinto

Mosh #343 My comment about the SRES was related to an earlier conversation about explicitly taking population, industrialization and urbanization into account when trying to model the anomaly trend to match reality, rather than just taking GHG out of the model that rests upon GHG to make it match. (When I take my engine out of my car, it doesn’t match the speed profile)

Ron #360 Thanks.

The other question (besides “Can SEM do the job?”) is “Why didn’t we use the standard deviation, or a mix of SD and SEM, or individual models?” and the answer is (as I understand it):

The IPCC doesn’t do it that way.

Or

A mix forces assumptions or actions beyond the scope of the test and would be far too uncertain.

Using the SD results in “error bars” (or whatever you want to call it) that are so large as to only prove “uncertainty” (or whatever you want to call it). We already know there’s uncertainty (since whe have to use an ensamble), and the purpose of an ensamble is to take models, regardless if they’re correct or not, to get an aggregate that is theorized to more closely be able to match reality.

About the same reasons as the other two; using individual models we know don’t match doesn’t help (because the results depend on which you pick), and why an ensamble is used in the first place. Looking at them individually also amounts to trying to validate them which is beyond the scope of the test.

Or

A mix or individually is beyond the scope of the test as well as being meaningless. The SD isn’t used mainly for that reason, as the purpose of an ensamble is supposedly to provide an entity to use the SEM on.

Or

We used SEM. You can do the same test and use anything you want to.

Larry T

My wife is 85lbs and i am 285lbs on the average we are only slightly overweight but that doesnt truely tell the story

Bob B

This argument doesn’t help make decisions on the farm!

http://sciencepolicy.colorado.edu/prometheus/archives/climate_change/001420teats_on_a_bull.html

steven mosher

re 365. TOBs

Kenneth Fritsch

So under beaker logic a model that fits the data better has nothing to reccommend it.
A model with GHG forcing fits better than one without GHG forcing. Under beaker logic
this is immaterial, so the logic used to reject douglas is applicable to attribution studies.
Now, dont get me wrong. I think beaker has the stronger argument ( SE versus SD) HOWEVER,
how does that argument play in attribution studies. Good question I think. I dont know.

Steven Mosher, I would not call Beaker’s POV, as expounded here at CA, beaker logic, because what he is stating is the generally accepted view in modeling statistics (as well as I have been able to determine). In fact many comparisons do not even use SD but instead merely present a range of computer outputs and the observed and then make subjective statements about it much as was the case where Douglas et al. complained about Karl et al. doing this for a nearly identical comparison to Douglas et al where they use SEM. I have read papers that appear to judge the state of the statistical testing in model to the observed rather subjective and to be saved by using Bayesian methodology to put it on a more formal footing.

I think if one pushed the beaker POV to its logical conclusion, use of SD (as also indicated by RC) for comparisons would fail also and in fact would show that a strict frequentist approach would always fail. It mostly comes down to the assumption that we have only a single realization of the observed and the observed realization has a chaotic content that means the mean Xmodel – Xobserved = E and not zero for a perfect model. Now I can see the logic of the Beaker POV given the assumptions he makes vis a vis the Douglas paper, i.e., if the comparison is the same as comparing climates modeled to observed and we cannot estimate E. That is not my point of contention. My point is that since Douglas use ratios of trends in their comparison I see the chaotic content of the observed and the models canceling since I am comparing two outputs from the same initial conditions and looking strictly at any GHG effects.

Please explain how and why you think beaker has a stronger argument about using SD in the place of SE for the Douglas case – without Beaker or someone else showing that the chaotic content remains in the observed.

The more I read about these types of comparisons in the climate modeling world the more I am convinced that the choice of statistical tests have much to do with what keeps the model outputs as being plausible and the reaction to the Douglas et al. using SEM would be expected.

steven mosher

re 369. I’m just calling it beaker logic to give him some credit for sticking to his guns
and remaining civil. As an aside I tend to judge threads by the people they attract, and
who sticks with it. So, willis is always thoughtful, as are you, and when RomanM comes along
it’s icing on the cake.

I rather liked your approach. I thought few people got it. You always have a fresh way of
going at things ( like in the surface stations).

steven mosher

re 368. you can call me mosh or moshpit. no worries. Getting the data is hard. Even for ModelE they only make certain data available and they always filter it! For example with modele
data you can only get the average of the 5 runs per esemble, and they give it to you as a running mean. The IPCC has an online database, so gavin has promised that I can get access
if I ask nicely and write up a good explaination of why I need the data. I’ll resubmit my
request an see what happens. Then let you guys know how to get the data. Or you can all go try
and see what they say to you. If your interested ask And I post a link

Kenneth Fritsch

Re: #372

How is the observed chaotic content measured and quantified when there is only a single draw? I do not understand how this is done and therefore cannot understand how the two chaotic components (earth’s climate, modeled climate) can be said to cancel.

I believe Beaker’s reference to chaotic content (my terminology not his) was that it depends (very sensitively) on the initial conditions and conditions that are not readily determined. We get only a single rendition of the observed so that we cannot average out that content using several runs.

Having said all that I am not saying that the cancelling of the chaotic content is between observed and models but between, for example, the surface and troposphere temperatures when one uses a ratio of surface to troposphere since both are realizations from the same initial conditions. I also know that the noise level of the climate is nearly the same in the surface and troposphere, i.e. they correlate well. I am therefore saying that since we are not comparing complete climate system, model to observed, but ratios of trends, models to observed, that the chaotic content does not come into play.

That has been my ongoing question to Beaker, who to this point has not responded.

If as Beaker suggests (and as do authors I have read on this subject)that the chaotic content depends on the initial conditions, I assume that one would make model runs with a range of initial conditions to get an estimate of the effect. Once one has this estimate why could not it be applied to observed in making comparisons.

Kenneth Fritsch

Re: #376

Sam, those three models passed a test proposed by Douglas in matching certain climate observations but did not “pass” a test (2 SD per Willis in a post in this thread) when comparing them to the observed ratio of surface to troposphere temperature trends.

Also remember that Beaker, and in other instances, other climate modelers were hesitant to remove models for reasons that were not clear to me. Also Douglas et al. complained in their paper that Karl et al. had included some model results in their paper’s comparison of the models to the observed that appeared to be outliers in a physical sense. In the end, removing dubious models reduces the SD and makes a biased model ensemble average easier to reject as the same as the observed.

I think I see a thread running through all this. Do not use SEM as a method of comparison of an ensemble to observed and do not eliminate physical outliers from the ensemble — and the models will live to see the light of another day.

Geoff Sherrington

Re 351 UC

I defer to Dilbert’s 87. Brilliant.

Sounds like the meaning of Life, the Universe and all that. 37.

But what is the approved method to calculate if the 87 and the 37 derive from populations that overlap with various degrees of confidence?

Geoff Sherrington

Re # 323 beaker

You write –
He (the modeller) then changes his opinions on the physics of the climate and this is reflected in the next (hopefully more accurate) generation of models.

A scientist does not change his/her opinion of physics. If there is doubt about the physics, the modeller should not start on them. Physics is more formal than “Do gentlemen prefer blondes?”. The modeller must have authenticity for his physics. The next iteration of the modeller you mention might be more “precise” following changes to the emphasis put on the parts of the physics, but it can only be more “accurate” if wrong physics are deleted and correct physics substitued. You want to change F=ma because in your opinion it is wrong?

Ron Cram

Sam,
re: 374

We have had more than 20 years of observations and the models can run hundreds of years in a particular run. If the models are not matching the observations within the time period available, they never will.

I think it is time someone pointed out the obvious here. In 2007, Roy Spencer and co-authors published their observations on a newly discovered negative feedback over the tropics. They identified this as confirming the “Infrared Iris” effect hypothesized by Richard Lindzen. When the peer-reviewed paper came out, they mentioned that no GCM currently factors in this negative feedback.

It is entirely possible that if this feedback was fed into the GCMs, they may produce results more in line with the observations. That still would not mean they have any predictive power, but they would mimic current climate changes better.

DeWitt Payne

Geoff,

That’s really curious. I’d never heard that before. Did they change it for SH audiences in the recent movie version too? What about the BBC TV and radio versions? OT I know, but this sort of trivia fascinates me.

Geoff Sherrington

Re # 385 DeWitt Payne (concluded)
This is way OT and I mean no disrespect to David Douglass and co-authors,
but the number 37 came from here, now I have dug deeper. Monty Python & Holy Grail (I think the text still has indirect relevance or irreverence in the context of Climate Science)

King Arthur: Old woman.
Dennis: Man.
King Arthur: Man, sorry. What knight lives in that castle over there?
Dennis: I’m 37.
King Arthur: What?
Dennis: I’m 37. I’m not old.
King Arthur: Well I can’t just call you “man”.
Dennis: Well you could say “Dennis”.
King Arthur: I didn’t know you were called Dennis.
Dennis: Well you didn’t bother to find out did you?
King Arthur: I did say sorry about the “old woman”, but from behind you looked…
Dennis: What I object to is you automatically treat me like an inferior.
King Arthur: Well I am king.
Dennis: Oh, king eh? Very nice. And how’d you get that, eh? By exploiting the workers. By hanging on to outdated imperialist dogma which perpetuates the economic and social differences in our society.

Peter Thompson

Beaker #388,

This certainly is the nub of the issue:

You need to read the IPCC comment someone posted. They claim that the ensmeble mean gives the best estimate of the “forced climate change” not of the observed climate (not the same thing).

What the IPCC unfortunately does not also do, is go on to say that since this post hoc tuned, flux adjusted mess of spaghetti covers every eventuality from virtually no temperature effect to Hades on Earth, we have no scientific physical basis for our claims concerning anthropogenic climate forcings.

Michael Smith

Beaker wrote:

Michael Smith #340: The jump from (iii) and (iv) to (v) is not a non-sequitur, if you use a test that will reject a model that has a meaninglessly small difference to the observations, it is a meaningless test.

If a test rejects a model with a meaninglessly small difference to the observations under all possible conditions — or even under typical conditions — then you can argue that it is a meaningless test under all possible conditions, or even under typical conditions.

Your argument, however, amounts to the opposite: namely, that a test that can be shown to be meaningless under any conditions is meaningless under all conditions. But you have not shown why this should be so. And the fact that in Douglass, the ensemble failure has nothing to do with the size of (n) demonstrates it to be false.

The difference between the ensemble and the observations in Douglass may not prove the two are inconsistent — as conventional statistics uses the term — but by your own admission that the test shows bias in the data, you cannot argue that the difference is meaningless.

beaker

I said:

The 2SE test is a test of whether we can be confident that the mean of the ensemble is the same as the observations or whether it is different

but that wasn’t quite accurate. The 2SE test is a test of whether the mean of the underlying population from which the ensemble is drawn is the same as the observations or whether it is different. As I have already explained we would expect them to be different from purely a-priori arguments. Thus if the ensemble passes the test it is only because the ensemble is too small for us to accurately estimate the population mean from the sample we have, and be sure that this difference does really does exist.
It is a test of something that we would not expect to be true and the ensemble passes if the uncertainties are too high to be sure of what we would expect. Great test! ;o)

beaker

Cliff Huston:

No. You are cherry picking quotes, which is consistent with the hand waving, I said you need to get past.

I’m sorry, but if you are going to take that attitude there is no point in discussing things any further with you. The IPCC quote appears to me to be worded that way to make an important distinction between what the models can predict and what they can’t. If you are happy to ignore such distinctions, it is not overly surprising that you don’t understand the point I am trying to make.

Accusing me of cherry picking is insulting, and unhelpful, especially when I have gone out of my way to be patient and civil with you.

beaker

steven: Cool, all we need to do now is to work out exactly what the question is ;o)

Ron Cram

beaker,
re: 388

You write:

Ron Cram #345: The real issue may be something else, that is fine. But if the issue is something else, then don’t try and support it with faulty statistical evidence. I would have no problem with a paper that just looked at the data nad drew a conclusion, it is the use of incorrect statistics that I have issue with.
If the statement (vi) is true, then it applies in all cases. If what you say is true, then all of the model runs would give exactly the same value for the trend, but they don’t.

I don’t understand which of my comments in 345 you are referring to in your first paragraph. Regarding your second paragraph: Nonsense. You are assuming that the actions of GHG gases are purely stochastic. That is contrary to the theory of AGW since GHG gases are believed to be deterministic when it comes to warming the troposphere faster than the surface. Surely you understand that by now. Climate is a non-linear coupled system so models would NOT “give exactly the same value for the trend.” (Which really is not the point anyway as this is about warming ratio of troposphere to surface.) Many other equations and parameterizations apply. But the models would have the troposphere warming faster than the surface. And indeed they do. The problem is not that the models do not match AGW theory. The problem is the models are not matching nature. Because these facts are true, statement (vi) cannot apply. And since statement (vi) does not apply, neither do any of your following statements.

Cliff Huston

#395 Beaker,
Sorry. You have my apology, no offense was intended. I will choose my words with more care.
Cliff

Christopher

Re: beaker Yeah, I tried to look at this by denying all assumptions/knowledge and turning it into an abstract statistical exercise. And I wanted get an actual probability but not in the SD vs SE context (that’s why I went nonparametric). I know of no test that can get at a chaotic component (the endgame metaphor). You have mentioned the SD test could but I don’t see how that could work. SD is also unbiased as n -> infinity (sans chaos). And since climate_sigma is in fact a “bias” you would miss it, no? Or, to turn it around, are you not making assumptions on the magnitude of climate_sigma? Saying that it’s “somewhere between” 2 asymptotic SE and 2 asymptotic SD? Put another way, if climate_sigma > 2 asymptotic SD then the same result ensues as with the vanilla SE test. Here are some numbers. I can generate 10000 standard normals. I get a sigma of 1.001 with whatever seed Matlab picked this instance, and an SE of 0.01. Fine. Let’s say I do a 2SD test. The test will fail (in your sense) if climate_sigma > 2.002? Yes? And the 2SE test fails if climate_sigma > 0.02. So I can have my perfect model but will still “miss” in both cases? It’s easier to “miss” on the SE test because to 0.02 will get ever smaller asymptotically. But you can “miss” on the SD test too. See, this is why I said that I am not interested in chasing the knowable. I’d love to have at it but if it is, by definition, truly unknowable then, well, I’m not sure what else there is to do. The other thing here is that you can condition on the ensemble to estimate climate_sigma. In my previous post the relevant CI was (13.45,285.25). So the observations are not in agreement with ensemble means provided that climate_sigma is

Kenneth Fritsch

Re: #388

Note that the ensemble predicts the climate with the effect of ENSO removed, but it is being compared with the trend for the observed climate where the effect of ENSO has not been removed. It is not a like-for-like comparsion, so there is
no reason to expect the ensemble mean to exactly match the observations.

Beaker, this observation gets to the crux of my proposition that since Douglas et al. uses ratios of trends (and of a considered GHG finger print at that) that the chaotic content is not in play. You have not addressed that proposition in any of your replies. I can show that the ENSO has no significant effect on the ratios of annual surface to troposphere temperatures and trends.

Sadly, you can’t really estimate E using the models, as you end up with a circular argument.

Which if true, and in the strictest meaning it is true, we have no consistent means of determining whether an individual model or an ensemble average is different than the observed. You have consistently ignored using an SE test where Xmodels – Xobserved = E, where E could be some reasonable estimate of the observed variable content. For practical purposes if E is large than the climate becomes essentially unpredictable to any reasonable limits. (Note that I am talking climate here and not what Douglas et al. were measuring).

Beaker, when you note that use of a SE can reject a perfect model I think that confuses what the SE test is actually testing. It is testing the average of an ensemble because we have no method of determining which individual model is perfect. That perfect model result (and not necessarily from a perfect model) could have occurred by chance.

Christopher

Still got cut: “less then 13.45.”

Kenneth Fritsch

Christopher @ #403:

I agree with your excercise and showing that if we take the strict view of Beaker and without a limit estimate on E, we have no way comparing model output to observed using SE or SD. If E is without limit then surely we conclude climate is not predictable, projectionable or estimateable. The models then become an excercise like solving a cross word puzzle with little practical value.

Karl et al. took a range of model outputs without an average when comparing the ratios of modeled tropical surface to troposphere temperature trends and then, according to Douglas et al., Karl made an observation that the observed was within the range of model outputs and thus could not conclude that model and observed outputs were different. In fact he concluded that results indicated that the observed data could be flawed.

To be fair to Beaker, I think his POV is in line with that of the climate modelling world. Remember that the IPCC and climate modelers started making projections of models to the future where there is no observed for comparisons. They could always fall back on a show of hands by the experts to put some uncertainty limits on those projections. Obviously a comparison of the observed to the model simulations gets more complicated. From my reading of the IPCC AR4 reports some authors push for use of an ensemble average but as I recall do not provide any means of statistically comparing it to the observed other than an show of expert hands.

steven mosher

By moshersteven at 2008-05-09

Cliff Huston

#395 Beaker,
You posed the perfect model ensemble, with an infinite number of member models to show that SE error bars would goto zero. I posed the perfect model ensemble, with a finite number of perfect member models to show that the SD error bars would go to zero. In both cases we have undoable mind models. To apply to Douglass et al, ENSO like effects are removed in both cases. The only chaos elements involved in judging against real world observations are those of the real world. The case for running an ensemble of perfect models with the same initial conditions in no sillier than running an infinite number of imperfect models to produce a perfect ensemble. Both cases are straw-man arguments, neither provide any useful insight into the correctness of using SD or SE in Douglass et al.

I keep harping on the IPCC intent of the model ensemble, because if the ensemble average is not the ensemble output, there is no useful prediction made by the ensemble. If there is no useful prediction from the model ensemble, there is no way to judge the theory built into the models. The use of SD to judge the ensemble members is, to my mind, valid – that is: are the member models consistent? But to use 2SD as the ensemble prediction is to use, potentially, the worst allowable model data as the prediction. The IPCC claims observation consistency only with the model ensemble average, which they call the best estimate, and make no claims regarding the ensemble model cloud. The methods used in Douglas et al are consistent with the IPCC’s claim, in that they state they are using the best estimate model data.
Cliff

steven mosher

Ok getting ModelE data

Start here http://data.giss.nasa.gov/

See the link for climate simulations of 1880-2003. click that

http://data.giss.nasa.gov/modelE/transient/climsim.html

Here you will see the link to the paper and all the readme I know of

now to get the data look at table 1.

See line #4. ALL FORCINGS. These are the similautions that include all forcings
( line 1-3 contain individual forcings, like GHG only for example, or volcano)

On the left hand side of the table you will see links for the forcing. On the RIGHT
you will see a list of RESPONSES.

select “lat time”

http://data.giss.nasa.gov/modelE/transient/Rc_jt.1.11.html

Now you will see a pull down menu.

See the first box. Quantity? Pick surface temp ( there are others as well )

Mean Peroid. pick 1 month to get rid of the running mean

Time interval: pick what you like.

base peroid. I selected 1961-1990 because I wanted to compare ModelE to hadcrut.

Output. Formatted page with download links

Show plot and then get the data

http://data.giss.nasa.gov/work/modelEt/time_series/work/tmp.4_E3Af8aeM20_1_1880_2003_1961_1990-L3AaeoM20D/LTglb.txt

MarkW

Two of my favorite mangled sayings, that I’ve run across on the web.

two-headed sword
and
pitch quiet

Richard Sharpe

Two of my favorite mangled sayings, that I’ve run across on the web.

two-headed sword
and
pitch quiet

I’m kinda partial to:

tow the line

and

bare with me

Christopher

>So why even bother trying to reconcile them, there’s no test they can pass.

Well, I do not believe this. Climate_sigma is overrated and there are tests that circumvent SD vs SE (I performed one earlier up). I think a very good test could be had but implementing a grid of parameter values and initial conditions across the current state-of-the-art generation of GCMs. Such a “grid search” would likely entail 100s of runs on each GCM –a problem to be sure but I’m trying to tease out if such a test is achievable. In any event, the SAT trajectories for each run by each GCM form the raw data. And then you apply a fairly complicated bootstrap to get a probability. This is doable and I would trust the result as the irreducible climate sigma is factored out. It’s in each individual SAT trajectory and it’s in the observations. You circumvent the “n thing” by comparing a bootstrap sample to an observed sample, one cannot swamp the other. It’s like doing the test I did earlier but generating your own critical values based on the bootstrap world.

Willis Eschenbach

AS always, reading everyone’s comments brings up interesting ideas.

Regarding model ensembles, It is worth noting the IPCC use of a “simplified” model. This model, called MAGICC (Model for the Assessment of Greenhouse gas-Induced Climate Change), can be tuned to reproduce the outputs of the various individual climate models, without the weather “noise”. The UN IPCC uses it to give many of the outputs shown in their report. Here’s a quote:

By using such simple models, differences between different scenarios can easily be seen without the obscuring effects of natural variability, or the similar variability that occurs in coupled AOGCMs (Harvey et al., 1997). Simple models also allow the effect of uncertainties in the climate sensitivity and the ocean heat uptake to be quantified.

Curiously, it can reproduce the output of the various models with the use of only six parameters. Here are typical settings, alonge with the explanations, from here:

Table 9.A1: Simple climate model parameter values used to simulate AOGCM results. In all cases the mixed-layer depth hm=60m, the sea ice parameter CICE=1.25 and the proportion of the upwelling that is scaled for a collapse of the thermohaline circulation is 0.3, otherwise parameters are as used in the SAR (Kattenberg et al., 1996; Raper et al., 1996).

_______AOGCM___________F2x (Wm -2)_________T2x (°C)___________T +(°C)___________k(cm2s-1)_____________RLO_________LO|NS (Wm-2 °C-1)_
____GFDL_R15_a___________3.71_______________4.2_________________8_________________2.3________________1.2_________________1_________
_____CSIRO Mk2____________3.45_______________3.7_________________5_________________1.6________________1.2_________________1_________
______HadCM3_____________3.74________________3_________________25________________1.9________________1.4________________0.5________
______HadCM2_____________3.47_______________2.5________________12________________1.7________________1.4________________0.5________
____ECHAM4/OPYC____________3.8________________2.6________________20_________________9_________________1.4________________0.5________
______CSM 1.0______________3.6________________1.9_________________-_________________2.3________________1.4________________0.5________
______DOE PCM______________3.6________________1.7________________14________________2.3________________1.4________________0.5________

F2x – the radiative forcing for double CO2 concentration
T2x – climate sensitivity
hm – mixed-layer depth
CICE – sea ice parameter (see Raper et al., 2001a)
T+ – magnitude of warming that would result in a collapse of the THC
k – vertical diffusivity
RLO – ratio of the equilibrium temperature changes over land versus ocean
LO|NS – land/ocean and Northern Hemisphere/Southern Hemisphere exchange coefficients

Now, the existence of this model and its use by the IPCC brings up some very interesting issues.

First, it directly answers the question about the independence of the models. Since the individual models’ global temperature results can all be replicated by the MAGICC model, by changing only six parameters, it is quite clear that regarding the global mean temperature, the global climate models are not separate individual realizations. They are just variations of the basic MAGICC assumptions. They are nothing more than the results of a time-dependent function

Temperature = MagicFuntion(v1, v2, v3, v4, v5, v6, F)

where v1 through v6 are the six variables, and F is a time-dependent set of forcing values. Different results are in no sense independent, they are a result of the same function with different tunable parameters.

Next, it lets us understand the nature of a large ensemble of models. Since the models are all reproducible by varying those six numbers, we can in theory use the MAGICC model to give us a probabilistic distribution of all possible model results. All we need to do is to generate random numbers, perhaps using the mean and the standard deviation of those variables as given by the different GCM models. That will give us a probabilistic distribution of the results of all possible models for a given set of forcings.

Alternatively, we could use the range of values given by current models, and randomly generate values bounded by that range. This, of course, will give a wider distribution of outcomes. However, by the law of large numbers, their mean will not be far from the mean done the other way (provided that we include sufficient models to start with).

Next, from this universe of models, we throw away all of the realizations that are not within a certain distance of the historical record given the historical forcings. It’s the necessary (but not sufficient) first step. This gives us the universe of MAGICC-type climate models that can hindcast the historical climate, including all those that exist now.

It is worth stopping at this point to consider this ensemble of thousands of remaining different model results. A long-standing unexplained puzzle of the models is, why does an ensemble of models give better results than a single model? According to the IPCC, it should be more accurate (closer to the observations) than any single random realization. I would suggest the reason is that through our tuning procedures, we are drawing from a universe of models whose overall average is a good approximation to the historical record. Because of this, we would expect the average to do better than any given model at hindcasting the record …

Next, it is worth recalling the recent Kiehl study discussed elsewhere on this site. He states that:

A number of studies (Raper et al., 2001; Forest et al., 2002; Knutti et al., 2002] have shown that there are three fundamental climate factors, namely: the climate forcing, the climate sensitivity, and the efficiency of ocean heat uptake. Given these three factors one can calculate the evolution of the global mean surface temperature. All of the models applied to simulating the 20th century climate represent these factors in varying degrees of sophistication.

These three factors are among the six factors listed above,

There’s more issues here, but I’m going to bed.

w.

steven mosher

re 418, not sure what you want exactly, can you explain

Kenneth Fritsch

Re: #426

I would like to take the discussion where you are attempting to go with it, but I am afraid the operative Beaker response here will be: but is it a perfect match.

Here is some more practical insight into this phenomena below.

By the way, how would like to have Beaker calling balls and strikes if you were the pitcher in a baseball game.

The link below from which I excerpted the following seems pertinent to this discussion. Note my bolded emphasis in the first paragraph below.

Click to access 226.pdf

Also consider just one of the shortcoming of climate models cited by the IPCC (and listed earlier in this paper), their inability to reproduce the observed vertical temperature profile of the atmosphere. All climate models project that increased greenhouse gas concentrations should lead to the mid- and upper troposphere warming faster than the surface. However, data for the last two decades indicates that the troposphere has warmed at a considerably slower rate than the surface. In 2000, the National Research Council concluded that the differences in warming trends between the surface and troposphere were real, i.e., not the result of measurement errors, and that they were not adequately reflected in climate models.20 More recently, Chase, et al.,21 researchers at the University of Colorado, Colorado State University, and University of Arizona, examined whether the differences between observations and the outputs of four widely-used GCMs were caused by either forcing uncertainties, i.e., uncertainty in the effects of greenhouse gases, aerosols, etc. on climate; or by chance model fluctuations, i.e., the variability caused by the model’s representation of the chaotic behavior of the climate system. The authors found that neither of these factors explained the differences between model projections and observations. They further concluded:

Significant errors in the simulation of globally averaged tropospheric temperature structure indicate likely errors in tropospheric water-vapor content and therefore total greenhouse-gas forcing, precipitable water, and convectively forced large-scale circulation. Such errors argue for extreme caution in applying simulation results to future climate-change assessment activities and to attributions studies and call into question the predictive ability of recent generation model simulations, the most rigorous test of any hypothesis.22

The errors identified are in the fundamental equations in climate models, and relate to the water vapor feedback that is part of every climate model. Without this feedback, doubling the atmospheric concentration of CO2 would result in a global average surface temperature increase of 1.2°C. However, any increase in surface temperature will increase the rate at which water is evaporated and raise the average atmospheric concentration of water vapor. Since water vapor is a greenhouse gas, the result is a further increase in temperature. Climate models project that doubling the atmospheric concentration of CO2 would result in a global average surface temperature increase of between 1.5 and 4.5°C. This large range is due to the differences in the way the models handle the water vapor feedback. The increase in atmospheric concentration of water vapor also results in models projecting an increase in global average precipitation.

Building more elaborate models at this time is unlikely to address the errors identified by the Chase, et al. As the Marshall Institute argued in it comments on the CCSP’s Draft Strategic Plan: “… model development should proceed only as fast as theoretical understanding of the climate system and validation permit.”23 Water vapor feedback is understood in qualitative terms; greater quantitative understanding is now required. On-going scientific studies offer hope that more quantitative understanding of the water vapor feedback is achievable. For example, a recent paper by Minschwaner and Dessler24 discussed observations from NASA satellites and scientific analysis indicating that there is less water vapor in the upper atmosphere than assumed by some climate models. In other words, climate models overestimate the size of the water vapor feedback and therefore potential future temperature rise. These findings, if validated by additional studies and then incorporated into existing climate models, should reduce the spread between model outputs.

Cliff Huston

RE: 425 Steven Mosher,

I was thinking of something like this:
Plot @ 1890 Temperature Tend 1880-1890 (annual anomalies)
Plot @ 1891 Temperature Tend 1881-1891
Plot @ 1892 Temperature Tend 1882-1892
.
.
.
Plot @ 2007 Temperature Tend 1997-2007

The idea was to get a look at how much decade trend wiggle there is over time, in the observed and modeled. I was hoping that you could do that without too much trouble.

Cliff

Willis Eschenbach

kuhnkat, you are close above, but not quite correct. The actual quote was:

When I arrived in Fermi’s office, I handed the graphs to Fermi, but he hardly glanced at them. He invited me to sit down, and asked me in a friendly way about the health of my wife and our newborn baby son, now fifty years old. Then he delivered his verdict in a quiet, even voice. “There are two ways of doing calculations in theoretical physics”, he said. “One way, and this is the way I prefer, is to have a clear physical picture of the process that you are calculating. The other way is to have a precise and self-consistent mathematical formalism. You have neither.”

I was slightly stunned, but ventured to ask him why he did not consider the pseudo-scalar meson theory to be a self-consistent mathematical formalism. He replied, “Quantum electrodynamics is a good theory because the forces are weak, and when the formalism is ambiguous we have a clear physical picture to guide us. With the pseudo-scalar meson theory there is no physical picture, and the forces are so strong that nothing converges. To reach your calculated results, you had to introduce arbitrary cut-off procedures that are not based either on solid physics or on solid mathematics.”

In desperation I asked Fermi whether he was not impressed by the agreement between our calculated numbers and his measured numbers. He replied, “How many arbitrary parameters did you use for your calculations?” I thought for a moment about our cut-off procedures and said, “Four.” He said, “I remember my friend Johnny von Neumann used to say, with four parameters I can fit an elephant, and with five I can make him wiggle his trunk.” With that, the conversation was over. I thanked Fermi for his time and trouble, and sadly took the next bus back to Ithaca to tell the bad news to the students.

Freeman Dyson, http://www.nature.com/nature/journal/v427/n6972/full/427297a.html

w.

Willis Eschenbach

Ron, thanks for your comments. You say:

First: Is this model used in attribution studies? If the model is designed only to predict climate change on the basis of greenhouse gases, isn’t this circular reasoning? Where is the model that seeks to understand natural climate variability? The model where you can tune TSI, PDO, ENSO, etc? Isn’t the model assuming zero natural climate variability? How can they say the temp record cannot be explained by natural climate variability alone if they have never tried?

This is several questions. AFAIK the model is not used in attribution studies. This model is only used because it can model the GMST (global mean surface temperature) output of other, much more complex models, in a very short time (compared to how long it takes to do model runs).

There are precious few models that seek to understand natural variability. This MAGICC model does not assume zero natural variability, it only models other models output without their inherent variability.

Modelers claim the warming at the end of the last century can’t be explained by natural variability because of the following (deeply flawed) logic, which runs like this:

1) Tune a model with all (or more likely some subset) of the known forcings to match the historical record.

2) Pull out the anthropogenic forcings, and note that the recent years no longer match the historical record.

3) Declare the case closed.

I find the recent study saying that we won’t see warming for twenty years unintentionally quite hilarious in this regard. After all the claims that natural variability could not be the reason for the late 20th century warming, guess what they ascribe the current cooling to … you guessed it, natural variability. And not only that, they ascribe it to a particular natural variability (the PDO) that is known to have caused warming for the last thirty years …

Second: The T+ parameter strikes me as odd. Is the only variable the “magnitude of the warming that would result from a collapse of the THC?” Can you not also determine at what level of warming the THC would collapse? It seems a great deal of uncertainty around that point is glossed over.

You have mis-read the citation, which says: “T+ – magnitude of warming that would result in a collapse of the THC” (emphasis mine).

As I mentioned above, Kiehl shows that the modeling can be done with three variables: forcing, sensitivity, and ocean heat uptake. The T+ parameter is a part of the third variable, it describes how the ocean uptake varies with temperature.

w.

Sam Urbinto

Does it matter exactly how the bits of metal fly out of the explosion to know they do?

Michael Smith

Re: #427

I would like to take the discussion where you are attempting to go with it, but I am afraid the operative Beaker response here will be: but is it a perfect match.

He may well ask that question. But I’d like to hear him explain how natural variability can account for the fact that the surface – troposphere relationship is the opposite of what AGW theory requires — not just what it predicts, but what it requires. (I think you’ve been making esentially the same point without getting a response, so I may not get one either.)

In the first place, the surface and troposphere observations generally move together, and, as I said in 426, there is simply no evidence in that data that shows internal variability having a long term effect (i.e. one lasting more than a month or two) on one of the two without also affecting the other.

Second, even if there is some natural variability suppressing troposphere temperatures without a corresponding affect on surface temperatures, that must mean that the surface heating that has occurred as a result of CO2-induced global warming can only account for a portion of the observed surface heating; it must mean that some other (non-CO2) factor has contributed to a significant portion of the surface heating — and that climate sensitivity must be lower than the models predict.

In either case, I see no way to invoke internal variability to save the ensemble from the conclusions reached by Douglass.

Cliff Huston

Well, it worked in preview – one more try:

Kenneth Fritsch

What I have been able to detect in my reading of background information pertinent to the subject of this thread appears as a rather repetitious and perhaps anticipated thread running through the ideas expressed in publications of comparing climate model output to observed results. It is expressed well in the paragraph of an article written by authors who are not antagonistic to the consensus view of AGW. The phrase “climate scientists have resisted statistical expression of uncertainty of climate model projections” says a lot and I think explains some of the reaction of bloggers/scientists at RC objecting vociferously to the rigorous Douglas et al. testing of the ratios of troposphere to surface temperature trends.

Click to access paper06.pdf

The first excerpt below summarizes the progression of expressing uncertainty of climate models and the components of it. The authors are proposing a Bayesian approach to comparing observed and model climate and using it to evaluate model variations in projecting future climate.

Note that the second excerpt below points to a rather obvious and critical limitation in their approach: the evaluation depends on cross validation of model outputs and while they claim to answer the skeptical claim that variation within the models invalidates them, the authors admit that “if there were systematic errors affecting future projections in all the GCMs, our procedures could not detect that”.

Until recently, most climate scientists have resisted statistical expressions of uncertainty in climate models. IPCC (2001, page 2) recommended expressing the uncertainties of climate model projections using phrases such as \virtually certain” (greater than 99% chance that a result is true), \very likely” (90{99% chance), and so on, but gave no guidance as to how these chances were to be assessed. Even this was considered an advance on previous IPCC reports, which made no formal acknowledgement of the role of uncertainty.

Nevertheless, over the past few years there has been growing recognition of the need for more rigorous statistical approaches. In this section, we summarize some of the main developments. Uncertainties in climate change projections are broadly of three types, (a) natural climate variability, (b) uncertainties in the responses to climate forcing factors, such as changes in atmospheric levels of greenhouse gases and sulfate aerosols, and (c) uncertainties in future emissions of greenhouse gases and other factors that could influence climate. The first two types of uncertainty are typically assessed in \detection and attribution” studies, which calibrate climate models based on their fit to existing observational data and which attempt to decompose observed changes into components associated with greenhouse gases, aerosols, solar fluctuations, and other known influences on the earth’s climate.

There are of course some limitations to what these procedures can achieve. Although the different climate modeling groups are independent in the sense that they consist of disjoint groups of people, each developing their own computer code, all the GCMs are based on similar physical assumptions and if there were systematic errors affecting future projections in all the GCMs, our procedures could not detect that. On the other hand, another argument sometimes raised by so-called climate skeptics is that disagreements among existing GCMs are sufficient reason to doubt the correctness of any of their conclusions. The methods presented in this paper provide some counter to that argument, because we have shown that by making reasonable statistical assumptions, we can calculate a posterior density that captures the variability among all the models, but that still results in posterior-predictive intervals that are narrow enough to draw useful conclusions.

Cliff Huston

Here is Douglass et al data including the outliers at 1000 kPa:

A lot of model ‘weather’ at 1000 kPa, we don’t want to lose that ‘information’. 🙂

Cliff

Cliff Huston

Oops, kPa -> hPa

Cliff

Cliff Huston

#446 Sam,
You say:

It appears surface to 925 is totally wrong, and from 200 and above very bad. Or am I misreading it?

I don’t know if surface to 925 hPa totally wrong, but by my count, 10 of the 22 models are not well at 1000 hPa and you are right that above 200 hPa the model trends are way too high – way beyond even 2SD.

I see both SEM and SD saying the same thing basically; SEM says “They’re way far away from the truth” and SD says “I have to have a huge plus-minus to get all of these in here”.

The reason I wanted to post plots of the Douglas et al data is that it is obvious that the wide 2SD is a function of the models not agreeing with each other. Beaker argues that the model spread is model ‘weather’, which ignores the fact that Dogulass is using decade trends that average out shot term weather effects. Also, Beakers argument ignores the fact that the models in the ensemble are different programs, using different secret sauce to simulate the climate. Back in the Troposphere thread I wrote to Beaker:

I think if you spend a little time studying this document: An Overview of Results from the Coupled Model Intercomparison Project (CMIP) ( http://www-pcmdi.llnl.gov/projects/cmip/overview_ms/ms_text.php ) you will find that what you are calling ‘simulated weather’ is actually only a difference between models.

Here is a really short version of what I wanted Beaker to see.

From the CMIP introduction:

In 1995 the JSC/CLIVAR Working Group on Coupled Models, part of the World Climate Research Program, established the Coupled Model Intercomparison Project (CMIP; see Meehl et al. 2000). The purpose of CMIP is to provide climate scientists with a database of coupled GCM simulations under standardized boundary conditions. CMIP investigators use the model output to attempt to discover why different models give different output in response to the same input, or (more typically) to simply identify aspects of the simulations in which “consensus” in model predictions or common problematic features exist.

Details of the CMIP database, together with access information, may be found on the CMIP Web site at http://www-pcmdi.llnl.gov/cmip/diagsub.php . The first phase of CMIP, called CMIP1, collected output from coupled GCM control runs in which CO2, solar brightness and other external climatic forcing is kept constant. (Different CMIP control runs use different values of solar “constant” and CO2 concentration, ranging from 1354 to 1370 W m-2 and 290 to 345 ppm respectively; for details see http://www-pcmdi.llnl.gov/cmip/Table.php ) A subsequent phase, CMIP2, collected output from both model control runs and matching runs in which CO2 increases at the rate of 1% per year. No other anthropogenic climate forcing factors, such as anthropogenic aerosols (which have a net cooling effect), are included. Neither the control runs nor the increasing-CO2 runs in CMIP include natural varations in climate forcing, e.g., from volcanic eruptions or changing solar brightness.
CMIP thus facilitates the study of intrinsic model differences at the price of idealizing the forcing scenario.

Fig. 1. Globally averaged annual mean surface air temperature (top) and precipitation (bottom) from the CMIP2 control runs.

Fig. 20. Globally averaged difference between increasing-CO2 and control run values of annual mean surface air temperature (top) and precipitation (bottom) for the CMIP2 models. Compare with Fig. 1, which gives control run values.

The model wiggles in the figures are from internal weather effects and it is clear that these effects would be lost in a decade trend. It is also clear that there is disagreement on what constitutes a correct physical model, shown by the 11.5C to 16.5C range in Fig. 1. The IPCC deals with this model disagreement by using the average of the ensemble, as their best estimate data, to compare with the observed – some sort of ‘wisdom of the crowd’ theory I guess. Given the IPCC’s position that the ensemble output is the average of the ensemble models, the obvious choice for the output error bars is SE.

Beaker argues that this model disagreement is somehow information, that is lost if SE is used and can only be preserved by using SD, so I guess he also believes in crowd wisdom, but he goes beyond the IPCC in that he believes that all crowd members are equally wise. In Beaker’s view the ensemble is an information cloud that is reconciled if there is greater than 5% probability that any of the cloud members could produce the observed.

The problem with Beaker’s argument is that the ensemble average is not only the IPCC’s best estimate, but it also follows best the GHG theory of what should be happening in the troposphere. The ensemble model members that are closest to the observed data, don’t match the GHG theory. If the model ensemble were reduced to the 10 models that best match each other, the ensemble average would be very close to the 22 model ensemble average, but that 10 model ensemble would fail against observation even using 2SD. The 22 model ensemble is only reconciled, using SD, because it contains flawed members.

Cliff

Cliff Huston

RE: 448 Sam,

Just to follow through on my ‘best 10 models’ speculation, I removed the 6 worst models above and the 6 worst models below the 22 model ensemble average. The remaining 10 models are those that best agree with each other. Here is the result:

As I guessed, there is little change in the ensemble average and except in the ensemble’s strange surface to 925 hPa region, there is little support for observation reconciliation.

Cliff

Raven

beaker says:

As I said, consistency is a weak test. If you wan’t to test the models to see if they are useful, then stop talking about consistency and come up with a reasonable definition of “useful”.

The trouble is the modellers keep making the assertion that data is “consistent” with models and that proves that the models are useful. This leads to arguments about the meaning of the word “consistent”.

Ron Cram

Mosh,
re: 451

I don’t know what you see in Gavin’s post that you think is insightful. It is another PR post, an attempt to salvage the reputation of the GCMs and AGW. In the past he has said ten years without a new high global temp would be reason to question the models. Now he is changing the standard to an “Unambiguous New Record” because it gives an extra eight years of funding to the AGW cause. He is using the fact the models are all over the place to argue for more time.

John F. Pittman

#449 Beaker
Thank you for your patience and help. I know you have asked several times about a test and a definition of useful. I think you may have answered this. If so, please, just direct me towards the answer. Also, it may simply be that I have “the cart before the horse”.

The usefullness has been defined by IPCC and climatologists that a natural climate response could not be found that explained the 20th and present warming; it is thus CO2. So I pose a test that is “Do the models or the temperature anomaly data itself support this definion of usefullness?” If it is indeterminate that would falsify the claim. In other words we have the same broad window of comparison for usefullness of historical temperature anomalies including their uncertainties and errors that we have for the models including their uncertainties and errors. If one cannot conclude this with information at hand (it is unknowable), then is not the claim suspect anyway?

Kenneth Fritsch

Re: #449

I see where Ground Hog day was postponed from Monday to Wednesday this week.

Using SD to determine whether the observed is part of the distribution of all the model results is the same as treating the observed as another model result, i.e. the observed has no added value as a target that the models are attempting to reproduce.

Not addressing the issue of the using the ratios of temperatures trends in order to effectively eliminate the effect of the chaotic variable to the observed (and for the model results also) by simply repeating that the SE cannot be used unless the observed matches the modeled mean exactly will get us to another Ground Hog day without much resolution of the question at hand. SE comparisons are made everyday knowing full well that the averages being compared are never exactly going to match. Think of a control chart process whereby the average of a well controlled process is estimated and limits are set to using the sample size for the control test to establish SE limits for the process. One would not expect that the average of the distribution of an in-control process was captured exactly by the original estimate.

To see through to the essence of these arguments on how to compare observed and model results it is instructive to compare how Karl et al. looked at similar model out puts to the observed for the ratios of surface to troposphere temperature trends in the tropics as Douglas et al. analyzed. Karl looks at the entire range of values for model results and says that the observed falls within that range and therefore that one cannot say the models disagree with the observed. In fact Karl uses model outputs that show outlier tendencies for the surface temperatures and concludes that using their comparison approach that the observed results are probably wrong.

One can look at the model results this way, but then one is forced to conclude that the model range that the authors apparently legitimize by using as a range in their comparison with the observed is so large as to be practically worthless. One can invoke an unbounded limit for a chaotic content of the observed as making an observed to mean model result impractical, but that in turn implies that a large or unlimited chaotic content in the observed makes climate projections/predictions unrealistic to impossible.

On the other hand, I would suspect the thinking behind the Douglas approach goes something like:

We have a GHG finger print that the models show in a very biased pattern as the ratio of temperature trends in the tropics whereby the troposphere warms faster than the surface. We have model results that show a reasonable rendition of the surface temperature trends with the observed and they all show more warming than the observed shows for the troposphere. We have data that shows that the random climate variations match well between the surface and troposphere temperatures. We want to compare the surface observations as a target for the models but still recognizing that the observations have a variable content. The models have a biased tendency as a whole or ensemble that most likely results from the GHG fingerprint. We want to compare the average model output for this feature of the climate prediction to an observed target.

beaker

Michael Smith #458:

(i) The SE test will reject a model with finite difference between the observed climate and the ensemble mean if the ensemble is large enough, because of the 1/sqrt(n) factor. This is true no matter how small the difference actually is.

(ii) The models do not attempt to model the observed climate, only the forced component of the observed climate, i.e. what is left after the effects of things that the models can’t predict (e.g. ENSO), however small they may be, have been averaged out.

(iii) As a corollary of (ii), we would expect there to be a finite difference, even if the model is perfect, as the observations do not represent the forced climate that the modellers wish to estimate using the ensemble mean.

(iv) As a result, the SE test will reject a ensemble that gives perfect predictions (in terms of what the models actually aim to do) purely on the basis that it is “too large”.

(v) as a consequence of (i) the SE test will fail a model that has an arbitrarily small difference from the observations purely on the basis that it is “too large”

See the point I am making, whether a model passes the SE test doesn’t depend purely on how closely it agrees with the observations, therefore it isn’t a sensible test of whether the models are in good agreement with the data.
There, I haven’t mentioned infinity anywhere.

Lastly, here is some food for thought: David Douglass says

We noted that previous papers and IPCC reports had introduced the concept of averaging over models. There is no scientific justification for doing so. The models are not independent samples from a parent population because the models are, in fact, different from each other in many ways.

The SE of the mean gives the uncertainty of estimating a population mean using a finite sample. Douglass apparently doesn’t believe the sample are representative of a parent population, so why is he using a statistical test based on its mean?

The SD test on the other hand makes no such assumption, it is just a simplified Gaussian approximation of an empirical distribution.

beaker

Christopher #460: Indeed, MAGICC does suggest a view of an underlying population, even though it still isn’t an i.i.d sample from that population. The use of an ensemble is itself perfeectly reasonable without the models being an i.i.d sample (done all the time in machine learning, try a google scholar search for “ensemble diversity”). It did seem inconsistent ( ;o) ) for Douglass et al. to use a test that does require them to be an i.i.d. sample though!

Kenneth Fritsch

Below is a link to an article from NOAA published in 2005 that shows correct procedures for comparing climate observations to modeled results that avoids potential problems if the comparison is confined to using error bars. It is merely a standard statistical method using SEM in comparing of means and as such would not, in my view change the conclusions of the Douglas et al. paper. Note that SEM is used to compare means of observation and model outputs.

Click to access jrl0501.pdf

In contrast, the two-sample case is fundamentally different in that, in general, looking for overlap from two sets of error bars is not equivalent to the appropriate t test. Examples based on the data in Table 1 are used to illustrate the nature of the problem. Suppose we have finite samples of values of some quantity from both observed data and from a general circulation model (GCM). Estimates of the mean (X) and standard deviation (s) of the sample values can be made along with the uncertainty [standard error (SE)] of the estimated means. As is common practice, a confidence interval about the estimated means can be constructed by taking +/- twice the standard error. The intervals given in Table 1 are displayed graphically in Fig. 1 in the form of error bars.
For example 1, three sets of error bars are shown on the left side of Fig. 1 for the observations (O), GCM (G), and their difference (D). In this example, the observations and GCM have equal standard deviations. It can be seen that there is considerable overlap of the error bars from the observations and GCM. In such a case, a researcher would typically conclude erroneously that there is no statistically significant difference between their respective means. An alternate approach to the same problem is to apply a two-sample t test. Such a test has been applied and the corresponding error bars about the difference of the means (D) do not in-clude zero. Based on this test, the same researcher would conclude that there is a statistically significant difference between the means.

The reason for this apparent paradox can be understood by considering the relationship between the SE of the mean of the individual samples (observations and GCM) to that of the SE of the difference of their means. The crucial factor is that in the case of the two-sample t test, the SE of the difference is estimated by “pooling” the variances from the two different samples. It should be noted that while the two-sample t test is well founded in statistical theory, the use of overlapping error bars in the two-sample case is not.

The depiction of example 1 in Fig. 1 can be used to understand the crucial distinction between the two approaches. In order for significance to be declared using the overlapping error bars approach, one would have to move the means of the observations farther apart until the bottom whisker from the observations just touches the top whisker from the GCM. This can be expressed mathematically as:

X1 – X2 =/> cSE1 + cSE2

where X1 (X2) is the mean of the observations (GCM), SE1 (SE2) is the standard error of the observations (GCM), and c is a constant that determines the level of confidence. The two terms on the rhs of (1) represent the distances from their respective means to the end of the whisker (i.e., half the length of the confidence intervals). In example 1, c = 2 since the confidence intervals represent +/- two standard errors. Note that throughout this paper no distinction is made between population and sample parameters; it should be understood that estimates of various population parameters from an available finite sample are being used.

Reference

Warming world altering thousands of natural systems – 14 May 2008

The study, by an international research team featuring many members of the Intergovernmental Panel on Climate Change (IPCC), is a statistical analysis of observations of natural systems over time. The data, which stretch back to 1970, capture the behaviour of 829 physical phenomena, such as the timing of river runoff, and around 28,800 biological species.

Researchers led by Cynthia Rosenzweig of NASA’s Goddard Institute for Space Studies in New York created a map of the planet with a colour-coded grid showing how much different regions have warmed or cooled between 1970 and 2004.

They then placed each of the thousands of datasets on the map and determined whether they were “consistent with warming” or “not consistent with warming”. Trees, for example, might flower earlier in regions where the climate has warmed significantly.

In around 90% of cases where an overall trend was observed, it was consistent with the predicted effects of climate warming, the researchers report in this week’s Nature.

Ron Cram

Mosh,
re: 454

Okay, I misunderstood. I thought you were saying there was something new in Gavin’s post. I guess the fact he admitted the GCMs are all over the place is new. His attempt to change the standard was also new, but I don’t like changing the rules in the middle of the game.

Jon,
re: 456

It is pretty obvious, from a scientific standard – not a political one, that AGW catastrophism is in trouble. No new temp records since 1998 and a number of peer-reviewed papers explaining why this is so. And now the PDO has turned to its cool phase and we probably will not see any new records for about 30 years. Without changing the ground rules, one would expect the funding to study climate change would be drastically cut within a few years (funding cutbacks have to wait until the politics follows the science). By changing the ground rules to “unambiguous,” Gavin gets to claim they need another eight years of funding.

Cliff Huston

RE: 463 Ken Fritsch,

Thanks for the link to the Lanzante (2005) manuscript. Lanzante’s advice is not to use error bars at all, for hypothesis testing. Rather he suggests using a two-sample t-test. The author’s main objection to the use of error bars, is based on the work of Schenker and Gentleman (2001), which I have not been able to find free access to. I have found several papers that discuss aspects of Schenker and Gentleman (2001) and it seems to me that Schenker and Gentleman (2001) could provide further insight into the best way to judge the data in Douglass et al. Perhaps someone here does have access to Schenker and Gentleman (2001) and can provide us with some additional clues.

Googling on Schenker and Gentleman (2001), I found that climatologists are not the only ones confused by the use of error bars. The following is from a preprint of Belia, S., Fidler, F., Williams, J., & Cumming, G. (2005).
( http://www.latrobe.edu.au/psy/cumming/docs/Belia%20Fidler%20et%20al%20PM%2005rev.pdf )

Our aim was to study some aspects of the understanding of graphically presented CIs and SE bars by authors of articles published in international journals. We investigated the interpretation of error bars in relation to statistical significance. This may not be the best way to think of error bars (Cumming & Finch, 2001, 2005), but is worthy of study because of the current dominance of NHST and p values, and because greater interpretive use of error bars is unlikely unless the relationship with p values is understood. We also investigated researchers’ appreciation, when comparing two means, of the importance of experimental design—in particular whether the independent variable is a between-subjects variable, or a repeated measure.

We elected to study graphically presented intervals because we believe that pictorial representation is often valuable, and can “convey at a quick glance an overall pattern of results” (APA, 2001, p. 176), and because we agree with the advice of the APA Task Force on Statistical Inference (Wilkinson, et al., 1999): “In all figures, include graphical representations of interval estimates whenever possible” (p. 601). By ‘error bars’ we refer to the ambiguous graphic, two of which are shown in Figure 1, that marks an interval and may represent a CI, SE bars, or even SD bars. All error bars we use are centered on means (M), and n is the number of data values contributing to a mean. CIs are calculated as M ± tC x SE, where SE = SD/ n, and tC is the critical value of t, for (n – 1) degrees of freedom, for the chosen level of confidence, C. For us, this is 95%, implying that tC is close to 2. In all cases, SE bars are M ± SE.

To make an inferential assessment of a difference between two means, it may be best to consider a single interval on the difference itself (Cumming & Finch, 2005). For two reasons, however, we chose to study a comparison of intervals on the two separate means. First, it is common in journals to see figures showing separate cell means, sometimes with error bars, and, with such figures, assessing any difference requires consideration of intervals shown on the separate means. The Publication Manual includes two examples of figures of this type (APA, 2001, pp. 180, 182). Second, Schenker and Gentleman (2001) reported that in medicine and health science a rule of thumb is sometimes used for interpreting CIs on two separate means. The rule maintains that non-overlap of two 95% CIs on independent means implies a significant
difference at the .05 level between the means, and that overlap of the two CIs implies there is no significant difference. This rule is widely-believed, but incorrect, and refers to CIs on separate means. In fact, non-overlap of the two CIs does imply a significant difference, with p distinctly less than .05, but overlap does not necessarily imply there is no statistically significant difference at the .05 level.

Here is another paper that provides an additional view of the issues: Overlapping confidence intervals or standard error intervals: What do they mean in terms of statistical significance? ( http://www.insectscience.org/3.34/Payton_et_al_JIS_3.34_2003.pdf )

Abstract
We investigate the procedure of checking for overlap between confidence intervals or standard error intervals to draw conclusions regarding hypotheses about differences between population parameters. Mathematical expressions and algebraic manipulations are given, and computer simulations are performed to assess the usefulness of confidence and standard error intervals in this manner. We make recommendations for their use in situations in which standard tests of hypotheses do not exist. An example is given that tests this methodology for comparing effective dose levels in independent probit regressions, an application that is also pertinent to derivations of LC50s for insect pathogens and of detectability half-lives for prey proteins or DNA sequences in predator gut analysis.

One thing I note about these papers is that the authors don’t seem to understand Beaker’s point that a CI of 2SEM cannot be used for hypotheses testing – go figure. They all seem to agree with Lanzante’s statement:

Before beginning discussion of the problem, it is worth reviewing some of the types of error bars that are commonly presented, along with related terminology. Error bars may represent the standard deviation, the standard error, the confidence interval, or some other measure of error or uncertainty. While the standard deviation is a measure of dispersion of individual observations about their mean, the standard error is the standard deviation of a derived statistic, such as the mean, regression coefficient, correlation coefficient,etc. A confidence interval can be constructed about a sample statistic such that it contains the true population statistic with a specified probability(2) and thus can be used in hypothesis testing. This note is relevant to error bars that represent confidence intervals for hypothesis testing.

Cliff

Cliff Huston

RE: 470 Sam,

I was under the impression the ensemble was a derived statistic.

According to the IPCC it is – see the IPCC statements in post #394
Cliff

Cliff Huston

RE: 463 Ken Fritsch,

Cliff, it is important on reading the Lanzante (2005) manuscript to note that his point (which makes sense from a t test of the differences using SEM) is that using overlapping error bars for determining significance will always favor the case of no significant difference, i.e. when Douglas et al. use overlapping errors bars they would have been more likely to see no significant difference when one actually exists.

I agree. The papers I linked in #496, make that point clear and put some numbers around it. What I don’t find is any support for using 2SD for the CI, but even in that case this issue would apply.

Cliff

Sam Urbinto

#471 Cliff

Well, if the IPCC thinks the standard (systematic) error of the ensemble mean of the forced portions of the models are a good estimate, then:

To the extent that unrelated model errors tend to average out, the ensemble mean or systematic error {Tm} will be small, {T} will approach TF and the multi-model ensemble average will be a better estimate of the forced climate change of the real system than the result from a particular model.

Matches well with:

the standard error is the standard deviation of a derived statistic

And it appears that depending upon one’s interpretation on the appropriateness of this measure or not will decide if the SEM is a prudent way to compare the mean of a model ensemble of forcings only with the reality that encludes forcings and natural variability.

I have no opinion one way or the other (my post was pointing out this is indeed a derived statistic, but it makes no comment on the appropriateness of one test or another) but it appears that all of this is apples to cherrys. In that (as I’ve said) SEM can’t pass because the idealized realization is not idealized, disproving the IPCC’s stance if indeed that is their stance) and SD can almost always pass (since the outliers make the bars wider).

I’d say if I have a group of whatever and get a +/- .2 on it, then add outliers both directions that boost that to +/- 100, then what can’t I fit? The jist of this is that if -90 is the reality, and I throw away those outliers, I fail reality. But if I include the outliers, and +.1 is the reality, what does that tell me?

Nothing! 🙂

Willis Eschenbach

According to Lanzante, the proper test for significance is not whether the 2σ (more accurately 1.96 sigma) error bars overlap. It is whether the absolute distance between the means is greater than the square root of [ (1.96 * σ1)^2 plus (1.96 * σ2)^2 ], where σ1 and σ2 are the two relevant standard deviations.

IF one agrees with Gavin/beaker that the proper measure is the standard deviation rather than the standard error of the mean, then we find the following:

__hPA______________95%______________Difference_____________________
Altitude___________CI_______________Bet. Means__________Result_____

__1000____________0.01_________________0.15_________Not Different_
___850____________0.08_________________0.15___________Different___
___700____________0.15_________________0.18___________Different___
___500____________0.22_________________0.24___________Different___
___400____________0.20_________________0.27___________Different___
___300____________0.25_________________0.32___________Different___
___250____________0.30_________________0.37___________Different___
___200____________0.42_________________0.53___________Different___
___150____________0.39_________________0.33_________Not Different_
___100____________0.49_________________0.28_________Not Different_

If the difference between the means is larger than the 95% CI value, the two are significantly different. Note that these calculations are done with the raw data, not set to the same starting point, and not normalized.

Note that everywhere between the surface and 150hPa, the models and the datasets are significantly different even when we use Gavin/beakers curious method.

w.

beaker

Ralph Becket #464:

The justification for taking the mean of the models is that they are approximations to an hypothetical ideal model.

No, it is just that the mean of an ensemble is generally on average a better predictor than any of its component parts. There is nothing to guarantee that this is in any way an “ideal model”.

The SE of this mean gives you confidence intervals of how far off the sample mean is from the right value. With more models the SE tends to zero, as expected.

Yes, it is the uncertainty in estimating a population mean from a finite sample. Note that Douglass points out it isn’t an i.i.d. sample, which would make the SE an invalid estimate of this uncertainty, but nevemind…

The test of the mean of the models should surely be with respect to the SE *and* the error on the observations.

If you wanted to know whether there was a detectable difference between the ensemble mean and the observations, yes it would be a fine test. However, that is not finding out if the observations are consistent or inconsistent with the models, or even if there is a meaningful difference, so it doesn’t really tell you anything about the usefulness of the models.

As I have already pointed out, that since the model ensemble is only an estimate of the forced component of the climate, wh know before we start that we should expect there to be a difference, even if the model is perfect. Therefore it seems nonsensical to test for a difference that we know should be there. If a model passes the SE test it is only because the ensemble is too small for us to be sure that the difference we exepct to see is not a random sampling artifact.

If we had perfect observations (no error) and a perfect model (no error) and they differed then we would have to conclude that the perfect model was wrong (i.e., the model is not actually a perfect model of the process being observed).

No, because the models only predict the forced component of the climate, the observed climate also has a component due to things like ENSO that the models are not capable of predicting, even in principle.

The next step is to ask how great is the discrepancy. At least then you can say something about whether the model is in the right ballpark.

Actually, that is the first thing that Douglass et al. should have done instead of using a test for a detectable difference (which we know a-priori is almost certainly there anyway).

Cliff/Ken #463/469:

Whether you use error bars or a (slightly more sophisticated) t-test is not really the issue. The SE test is a test for a statistically detectable difference between the mean and the data. We know a-priori that a difference exists, so why is it any criticism of the models that the ensemble is large enough for it to be detected?

One thing I note about these papers is that the authors don’t seem to understand Beaker’s point that a CI of 2SEM cannot be used for hypotheses testing – go figure.

Except that isn’t the point I was making (of course). The SE can be used for hypothesis testing, just not the hypothesis that the models are inconsistent with the data, or that the models are useful. It can be used to test the hypothesis that there is a difference between the ensemble mean and the observations, but then that is what we would expect to see anyway, even if the model was a good one.

Kenneth Fritsch #472:

Beaker has an opinion about these tests just like RC does, but if one understands the underlying assumptions one can judge for oneself whether the beaker is half full or half empty. I know it is rather empty of references, links and details.

I have provided references where the SD error bars are used to test whether models are inconsistent with the data. You have provided references to papers showing the SE test can be used to detect a difference between the mean and the observations. Did Douglass et al claim to have detected a difference, or did they claim to have shown the models were inconsistent.

If you understood the underlying assumptions, you would know the SE test is only a test for a detectable difference, not a meaningful one (and as I said, we know the difference exists a-priori – so why test for it?).

Willis Eschenbach #477:

Have you used the individual model runs in calculating the SD or just the means for each model? If not the SD doesn’t include the stochastic uncertainty of the simulations, so it isn’t a test for inconsistency in the statistical sense. I have made this point several times. BTW it isn’t my curious method, I have given references to other papers using the same definition, and is what a statistician would recognise as inconsistency.

Not that consistency tells you if a model is useful, as I have also repeatedly pointed out.

Willis Eschenbach

Cliff, you say:

RE:477 Willis,
Looking at my plot of the Douglass Data in #439, I’m having trouble relating to your conclusion of ‘Not Different’ for 150 and 100 hPa. Either I screwed up the data plot or there is something else I don’t understand. The differences between the means on the plot look larger there, than anywhere else.

Cliff

Good catch, Cliff. The auditing never stops. You are 100% correct, my labels were reversed. So, they are not statistically different. Ignore Number 477, what was Nixon’s lovely phrase … “Those statements are inoperative”.

w.

Christopher

Re: Gavin on forced (see the what the IPCC models really say post)

“Claims that GCMs project monotonic rises in temperature with increasing greenhouse gases are not valid. Natural variability does not disappear because there is a long term trend. The ensemble mean is monotonically increasing in the absence of large volcanoes, but this is the forced component of climate change, not a single realisation or anything that could happen in the real world.”

So beaker is correct here. Steve would likely agree to that statement as well. I’m somewhat surprised we are still debating this. Again, there is no one way to resolve this mess as statistical significance (ill-posed or otherwise) does not equate to practical significance. I’ve stated that climate sigma is overrated and the reason is that I do not think it’s greater than the distance from the “observed” vs. “predicted” lines in any of Douglass graphs, SD/SE this way or that. But that is practical significance. Again, there’s no test for this. You can only threshold this to my knowledge. I’m hoping we can leave the wordsmithing behind and move forward, LTP and ergodicity as confounding factors here are more compelling imho. A pity bender left us long ago…

bender

Beaker is obfuscating the argument to try to win a trivial point. Yet the point he is trying to win only serves to further the relevance of the Douglass observation. The observed data do not fit within the expected* range of predictions. Of course the “expectation” may be wrong, and that is a separate issue. If internal climate variability is under-represented in the GCMs then the expectation is far from realistic. Is that what Gavin is arguing? He is unclear, and shifting about. As usual.

Kenneth Fritsch

Re: #484

Who here says use the error bars on 55 modeled (toy) results which are ensembled???

I sometimes miss the point that the poster intended in cases such as this one.

I reiterate: the paper referenced is pointing to the problem of missing statistically signifigant differences when using error bars instead of the proper t-test using means and SEMs. In effect it says that Douglas et al. were being too conservative in using error bars. The author also points to abuse of error bars in IPCC reports and the Journal of Climate. I would add that I have seen that abuse at CA also.

Sam Urbinto

Statistical mumbo-jumbo aside, what is the practical nature of a range of anything that takes into account any reading between -5 and +5 as being “consistent”?

Kenneth Fritsch

Re: #488

No, that is not the case as Douglass et al. ignore the uncertainty on the observations altogether, so they do not test whether the error bars overlap (which is common practice, but overly conservative as noted).

Below I have excerpted the following from Douglas et al. that hardly would indicate that the authors ignored the “uncertainty of observations altogether”:

3.2. Uncertainties

3.2.1. Surface The three observed trends are quite close to each other. There are possibly systematic errors introduced by urban heat-island and land-use effects (Pielke et al., 2002; Kalnay and Cai, 2003) that may contribute a positive bias, though these are estimated as being within ±0.04 C/decade (Jones and Moberg, 2003).

3.2.2. Radiosondes
Free et al. (2005) estimate the uncertainties in the trend values as 0.03–0.04 C/decade for pressures in the range 700–150 hPa. For HadAT2, using their figure 10, we estimate the uncertainties as 0.07 C/decade over the range 850–200 hPa. For RAOBCORE, Haimberger (2006) gives the uncertainty as 0.05 C/decade for the range 300–850 hPa.
Several investigators revised the radiosonde datasets to reduce possible impacts of changing instrumentation and processing algorithms on long-term trends. Sherwood
et al. (2005) have suggested biases arising from daytime solar heating. These effects have been addressed by Christy et al. (2007) and by Haimberger (2006). Sherwood
et al. (2005) suggested that, over time, general improvements in the radiosonde instrumentation, particularly the response to solar heating, has led to negative
biases in the daytime trends vs nighttime trends in unadjusted ropical stations. Christy et al. (2007) specifically examined this aspect for the tropical tropospheric layer and indeed confirmed a spuriously negative trend component in composited, unadjusted daytime data, but also discovered a likely spuriously positive trend in unadjusted nighttime measurements. Christy et al. (2007) adjusted day and night readings using both UAH and RSS satellite data on individual stations. Both RATPAC and HadAT2 compared very well with the adjusted datasets, being within ±0.05 C/decade, indicating that main cooling effect of the radiosonde changes were evidently detected and eliminated in both. Haimberger (2006) has also studied the daytime/nighttime bias and finds that ‘The spatiotemporal consistency of the global radiosonde dataset is improved by these adjustments and spurious large daynight
differences are removed.’ Thus, the error estimates stated by Free et al. (2005); Haimberger (2006), and Coleman and Thorne (2005) are quite reasonable, so that the trend values are very likely to be accurate within ±0.10 C/decade.

3.2.3. MSU satellite measurements
Thorne et al. (2005a) consider the uncertainties in climate-trend measurements; dataset construction methodologies can add bias, which they call structural uncertainty (SU). We take the difference between MSU UAH and MSU RSS trend values, ∼0.1 C/decade, as an estimate of SU.

Much has been made of the disparity between the trends from RSS and UAH (Santer et al., 2005) – caused by differences in adjustments to account for time-varying biases. Christy and Norris (2006) find that UAH trends are consistent with a high-quality set of radiosondes (VIZ radiosondes) for T2LT at the level of ±0.06 and for T2 at the level ±0.04. For RSS the corresponding values are ±0.12(T2LT) and ±0.10(T2). For T2LT, Christy et al. (2007) give a tropical precision of ±0.07 C/decade, based on internal data-processing choices and external comparison with six additional datasets, which all agreed with UAH to within ±0.04. Mears and Wentz (2005) estimate the tropical RSS T2LT error range as ±0.09. Thus, there is evidence to assign slightly more confidence to the UAH analysis.
UMD does not provide statistics of inter-satellite error reduction, and, since the data are not in a form to perform direct radiosonde comparison tests, we are unable to estimate its error characteristics. In the later discussion we indicate the likelihood of spurious warming. UMD data were not in a form to allow detailed analysis such as provided in Christy and Norris (2006) to generate an error estimate.

beaker

#491 There is evident uncertainty in the observations, as you get different values for the trend using data from different sources, as error bars were not placed on the observations, the uncertainty of the observations was ignored in the statistical test. Therefore it is biased in favour of rejection of the models rather than acceptance.

As I have commented earlier, it would be more productve to try to come up with a meaningfull definition of useful that we can test for rather than try an salvage the test used by DOuglass et al. which is meaningless (although that does not mean that their conclusion is wrong, just that it isn’t supported by the test they perform).

Michael Smith

Beaker, in 459 wrote:

(iv) As a result, the SE test will reject a ensemble that gives perfect predictions (in terms of what the models actually aim to do) purely on the basis that it is “too large”.

(v) as a consequence of (i) the SE test will fail a model that has an arbitrarily small difference from the observations purely on the basis that it is “too large”

There, I haven’t mentioned infinity anywhere.

Beaker, I know you’ve ceased being serious when you expect me to believe that switching from the term “infinite” to the term “too large” changes anything. Whatever you call it, your demand that any use of the SE test be based on the assumption of an unlimited “n” constitutes the assignment of an arbitrary constraint to that test.

Mike B

Beaker, I know you’ve ceased being serious when you expect me to believe that switching from the term “infinite” to the term “too large” changes anything. Whatever you call it, your demand that any use of the SE test be based on the assumption of an unlimited “n” constitutes the assignment of an arbitrary constraint to that test.

First, my apologies in my absence from this thread in refuting some of Mr. Beaker’s more tortured criticisms of Douglass. I’ve had other obligations…

Let me address Beaker’s latest argument (can’t use SE test because sufficiently large n will always reject) on two fronts.

First, Beaker has argued repeatedly that “we” need to focus on defining “useful”. Well, Beaker, if you would define useful, then your above concern would be rendered moot, because any statiscally significant(induced by sufficiently large n) difference smaller than the “useful” difference would be irrelevant. Goose meet gander.

Second, following your “too large” lead, as long as the between-model sigma (actually s) is too large, the SD test will never have the power to reject the hypothesis that the ensemble mean is different from the observed value. How convenient for Gavin and the other climate modelers! The more the models diverge, the harder it becomes to falsify the ensemble!

Douglass provides a devastating analysis of the efficacy of GCMs. Gavinian tantrums and Beaker’s carefully parsed nit-picking change nothing.

Cliff Huston

Beaker says:

As I have commented earlier, it would be more productve to try to come up with a meaningfull definition of useful that we can test for rather than try an salvage the test used by DOuglass et al. which is meaningless (although that does not mean that their conclusion is wrong, just that it isn’t supported by the test they perform).

While I strongly disagree that the test used by Douglass et al is meaningless, I do agree that a more robust presentation of their test is in order. Lanzante (2005) has shown that a t-test is more robust for comparing climate model ensemble data to observed ensemble data (link to the paper is in #463 above). Following Lanzante (2005), here is the unpaired t-test results for Douglass et al.

The above was calculated using the Web based unpaired t-test calculator found here.
It is clear that the t-test results are in complete agreement with the Douglas et al test and, in this case, that the extra robustness of the t-test was not needed.

Cliff

Craig Loehle

Cliff: thanks for the t-test. How come it took until post 500 for someone to do this?

Ron Cram

Cliff,

I also thank you. Perhaps you can publish this somewhere?

Geoff Sherrington

Still, I remain skeptical of model error estimates when the modeller does not state how many model runs were rejected voluntarily before submitting those published. I was taught that distributions help dictate the type of statistics to be used and that the whole population of relevant observations should be used to construct the distribution. Arbitrary or subjective selection of a sub-population is not scientific, is it? Because these are projections, there cannot easily be any criteria for selection/rejection, apart from a discovered violation of known physics/chemistry.

There remains a probability that one of the silly-looking rejected projections will, as the real results roll in over the years, be the one that was closest to reality – irrespective of the learned discussions of statisticians above. Sorry, I’ve seen it happen before in other fields. I suspect, but have no good evidence, that this is the story of the ozone hole as well.

Can we close this off now? The angels have danced on the head of the pin.

Cliff Huston

Craig, kim, Ron, Kenneth: Thanks for the kind words.

Craig:

How come it took until post 500 for someone to do this?

Mostly it has been all of the silly side issues due to the Gavin/Beaker spin. Until I saw the Lanzante (2005) paper, posted by Kenneth, I didn’t know that the t-test could be applied to this issue. Still I was unsure how to do the t-test until I read Roger Pielke, Jr.’s post “The Helpful Undergraduate”, then it was just a matter of turning the crank. Sorry to be slow, but I’m learning. 🙂

kim: Yes, but remember: bad pennies have their own odds.

Ron:

Perhaps you can publish this somewhere?

I have, here. 🙂 I think the t-test and a plot as in #494 would be a good addition to Dogulas et al’s SI. I wonder if that is still doable now.

Kenneth:

One should also remember that the paper was making a comparison to UMD, UAH and RSS data at two tropospheric levels of T2lt and T2 that would make the observed ensemble more difficult to present in this form.

I read that comparison as a first order reality check on the balloon data and their test was only using the balloon data as observations.

BTW, I decided to try using the unpaired t-test that does not assume equal variance in the samples. Here is the quick and dirty results from Excel:

Welch’s T-Test (unequal variance):
Surface___0.855783081___Not Significantly Different
1000 hPa__0.012353380___Significantly Different
925 hPa___0.008871460___Very Significantly Different
850 hPa___0.002147190___Very Significantly Different
700 hPa___0.000516788___Extremely Significantly Different
600 hPa___0.001357159___Very Significantly Different
500 hPa___0.000616360___Extremely Significantly Different
400 hPa___0.000063744___Extremely Significantly Different
300 hPa___0.000000809___Extremely Significantly Different
250 hPa___0.000000369___Extremely Significantly Different
200 hPa___0.000017286___Extremely Significantly Different
150 hPa___0.000000004___Extremely Significantly Different
100 hPa___0.000000630___Extremely Significantly Different

In case anybody would like to check my work or play with the data, I have zipped up my Execl spread sheet, along with a few of the plots as PDFs – the zip file can be found here.

Cliff

Ron Cram

Cliff,

Publishing it here is great and I thank you for it. I guess what I had in mind was possibly publishing it as a comment in the journal that published the Douglass paper. You could discuss the criticism from RealClimate and your analysis and conclusions. Making it part of the SI is an idea, but would be below the radar of most readers of the journal. I would encourage you to look into it.

Kenneth Fritsch

Cliff, for what it is worth having an amateur checking your calculations, I verified what you found in Post #499. I would go with this more conservative estimate over that used in Post #507, even though that means for the small sample case pooling the SD estimate when it is rather obvious that the observed data has a smaller SD than that for the models — at most of the pressure levels.

Kenneth Fritsch

Re: #512

Sam, I think one needs to be careful what one is referencing here. When I looked at how well the climate models matched observed past regional temperatures from the AR4 and AR3 reports, I found that the +/- variations, and biases, for that matter, were invariably large. I believe that is the same concept of absolute temperature to which you are currently referring to in your post.

What I also found from these reports was that when anomaly trends were reported and compared to the observed that the error bars decreased significantly. I have to assume – even though I found no explicit explanation in the reports – that the models have large errors in absolute temperature simulationsand smaller ones in hindcasting temperature anomalies.

All this leads me into taking this a step further with using ratios of temperature anomaly trends between surface and troposphere which should be easier to model (assuming that the correct theory is being applied) and easier to measure and particularly so for the satellite observations.

I am not defending the climate models here but simply observing that there are levels of measurements which the models can make and with which some are evidently less apt to result in large errors.

Sam Urbinto

That should have been above ~ -1 to +1

Thinking further, before CMIP (1995) they had to track it also, so probably a less sophisticated ensemble, or simply one of the ones around 14 C or something else…

Kenneth 513

I was simply making the observation that the “base period” (zero line for the comparison) seems to be a model ensemble output of some sort, which would make the anomaly a variation from some mathematical operation on a model or group of models. If so, it doesn’t seem too surprising that we’d move around in the range, nor be unable to fit inside 3 degrees of freedom so to speak. It’s like comparing observations to an ensemble, but almost sort of in reverse.

I have no issue with any of the comparisons, even if they’re “consistent but meaningless” or even “inconsistent but meaningful” or however it’s put.

Kenneth Fritsch

On first glance at the RC post, it would appear that the new paper is simply a rehash of the position looking at the models and assuming that the central tendency means something and making a valid comparison with the measured results as Douglass et al. did or taking the position of the authors that the model results are so varied that the range the model results can cover the measured results. The latter position could certainly support the position that the models are so uncertain that the results are meaningless vis a vis the ratios of the temperature trends between the tropical surface and troposphere.

It appears that RC employs the advocacy arguments of a good defense lawyer here, although I am not sure how the graphs of the model versus measurement results would fly with a jury that was either not committed to a conclusion or who are half-way intelligent.

hswiseman

Santer et. al 2008 never misses an opportunity to take a pot shot at Douglass, implying that Douglass’ use of the then current datasets was some kind of slipshod approach. I am sure that if Douglass had merely asked, they would have sent him an FTP password to access all the newly recalibrated data. The contempt for UAH also slips through, holding out RSS data as somehow far superior, when in fact the results are quite close, and with far less divergence than exists in the 49 models. I wonder with the slight difference between the two was enough to move the significance needle and render the conclusions slightly less robust. A result-oriented need to select one over the other might explain the intensive justifications offered.

I cannot comment on the claim of bias in Douglass’ screening technique for selecting eligible models, but there are alot of fancy statistics to support Santer’s filtering approach and claim of bias. Douglass’ approach was commonsensical and mechanistic, but that alone doesn’t mean it is without fault, Occam’s Razor notwithstanding. I really don’t see what is accomplished by the petty and contemptuous tone of the article, but that attitude seems pretty pervasive these days among the AGW crowd. Such an attitude isn’t that constructive here at CA either (maybe I should not have posted that monkey chart video), but this is blog-land, not publishing.

I think as a general principle that anytime you take a scalpel to the data in order to make the models look better, the burden of justifying each and every alteration falls heavily on the authors. I will leave it others with the math and stats chops to opine whether that burden has been carried.

Kenneth Fritsch

I have just finished a first read of Santer et al. (2008) that was published in reply to the Douglass et al. (2007) paper that had thrown into doubt the capability of the climate models to reproduce the the observed results of the ratios of the trends in tropical surface and troposphere temperatures over the recent past.

I think that the proper reaction to Douglass et al. and the recent Santer et al. should be a big thank to both parties. Thank you, Douglass et al. for provoking the reaction of Santer et al. and thank you Santer et al. for showing us that not only is the variation of the models from floor to ceiling but when you throw all the observations together using the more recent sources of measurements of the tropical troposphere temperature trends, we see that the observation variation is apparently from floor to ceiling (and maybe the sky).

If one is willing to wade through the obligatory advocacy oriented statements from Santer et al. as indicated from the following statement, “We find that there is no longer a serious discrepancy between modelled and observed trends in tropical lapse rates. This emerging reconciliation of models and observations has two primary explanations.” and proceed to a graph (shown below) that summarizes the model and observed results, one can readily conclude the following:

1. There exists a huge range in model results as indicated by the 2 Stdev of the model trends the reaches approximately 0.6 degrees C per decade.

2. An apparently even larger variation exists in the observed results as indicated by the individually displayed trends from 7 radiosonde and 2 satellite sources. Note that the radiosondes cover the entire pressure levels while the satellite results are confined to pressure zones. i.e. T2 and T2L.

3. On my first read I did not see any discussion of something that hits the layperson smack in the eye, i.e. the shapes of the observed results are very different than the average of the model results. Even the newer radiosondes measurement sources, that Santer et al. claims bring the observations more in line with the models since they show higher tropospheric temperature trends, only show that characteristic at some points on the pressure level range. In fact, all of them except RICH, show curves over the entire pressure level range that are considerably different then the model average and the older radiosonde measurements. As layperson I would suspect that a better statistical analysis of the observed versus model results over the entire pressure level range would be to compare the shapes of the curves for statistically significant differences.

I also have a problem with the graph in that the important value is the difference between (or ratio of) the surface and troposphere temperature trends and this graph does not show that value.

Finally the Santer paper makes reference to some recent correction of the RSS measurement that brings it closer to the model result average but further from the UAH measurements. I would like to read more about that adjustment and any comments of Spencer and Christy. Does anyone have a reference?

Figure 6. Vertical profiles of trends in atmospheric temperature (panel A) and in actual and synthetic MSU temperatures (panel B). All trends were calculated using monthly-mean anomaly data, spatially averaged over 20 °N–20 °S. Results in panel A are from seven radiosonde datasets (RATPAC-A, RICH, HadAT2, IUK, and three versions of RAOBCORE; see Section 2.1.2) and 19 different climate models. Tropical TSST and TL+O trends from the same climate models and four different observational datasets (Section 2.1.3) are also shown. The multi-model average trend at a discrete pressure level, _ bm(z) _, was calculated from the ensemble-mean trends of individual models [see Equation (7)]. The grey-shaded envelope is s{}, the 2σ standard deviation of the ensemble-mean trends at discrete pressure levels. The yellow envelope represents 2σSE, DCPS07’s estimate of uncertainty in the mean trend. For visual display purposes, TL+O results have been offset vertically to make it easier to discriminate between trends in TL+O and TSST . Satellite and radiosonde trends in panel B are plotted with their respective adjusted 2σconfidence intervals (see Section 4.1). Model results are the multi-model average trend and the standard deviation of the ensemble-mean trends, and grey- and yellow-shaded areas represent the same uncertainty estimates described in panel A (but now for layer-averaged temperatures rather than temperatures at discrete pressure levels). The y-axis in panel B is nominal, and bears no relation to the pressure coordinates in panel A. The analysis period is January 1979 through December 1999, the period of maximum overlap between the observations and most of the model 20CEN simulations. Note that DCPS07 used the same analysis period for model data, but calculated all observed trends over 1979–2004.

• Pat Keating
  Posted Oct 14, 2008 at 7:34 PM | Permalink

  Re: Kenneth Fritsch (#527),

  Where did the RICH data come from?

  • Kenneth Fritsch
    Posted Oct 15, 2008 at 9:26 AM | Permalink

    Re: Pat Keating (#528),

    Pat, below I have excerpted a comment from the Santer et al. (2008) paper describing and referencing the RICH dataset and measurements.

    The third (RICH; ‘Radiosonde Innovation Composite Homogenization’) uses a new automatic data homogenization method involving information from both reanalysis and composites of neighbouring radiosonde stations (Haimberger et al., 2008).

    I would strongly recommend the Steve M review of the radiosonde datasets and data manipulation in his thread at CA titled, “Leopold in the Sky with Diamonds” that is linked here:

    http://www.climateaudit.org/?p=3082

    Below I have excerpted a number of comments (in no particular order) from Steve M’s review that I think point to major concerns with the methodologies used and describe some of the essentials of the data manipulations.

    It is my view that given adjustable data some in the climate science community can adjust it rather arbitrarily without detailed and proven methodologies as evidenced by the wide range of the results of individual massages given by various scientists. The further upshot of this process is that the modelers can than point to their preferred manipulated data for comparisons.

    Raobcore v1.4 is hardly the last word in radiosonde adjustments. Leopold Haimberger (and I intend no slight by the title of the post, I just liked the sound of it) has already moved on to yet another adjustment system (“RICH”); Allen and Sherwood 2008 have used Iterative Universal Kriging, adding wind information into their adjustment brew. What each of these studies has in common is that none of them are new “experimental verification”; they are merely adjustments of ever increasing magnitude.

    Allen and Sherwood 2008 try a different tack – they try to create a homogenized wind data series on the basis that the radiosonde wind data is much less screwed up. They then argue that the trends in wind are consistent with tropical troposphere warming. They use this as evidence for the side of the argument that the UAH satellite temperature trends in the tropics are incorrect. I guess that we’ll see more about tropospheric wind data in the next while.

    So Raobcore v1.2 argued in a peer reviewed journal that there were post-1986 inhomogeneities in the ERA-40 model that required adjustment, giving a list of such inhomogeneities. Raobcore v1.4 decided that adjustments to ERA-40 after 1986 were not required after all. One would have thought that Journal of Climate would have required a detailed explanation of why Haimberger et al had changed their views so quickly and a detailed analysis of each post-1986 adjustment that was no longer deemed pertinent. A year earlier, Haimberger expressed concern about “jumps” at the end of NOAA-14. A year later, he was no longer concerned. Why? Is there any such analysis in Haimberger et al 2008? Nope.

    The algorithm in Haimberger et al 2007 added a novel tweak to changepoint methods – a tweak that should not be accepted a proven methodology, merely because it’s been published in a journal with weak statistical refereeing (Journal of Climate):

    “This paper introduces a new technique that uses time series of temperature differences between the original radiosonde observations (obs) and background forecasts (bg) of an atmospheric climate data assimilation system for homogenization.”

    • Pat Keating
      Posted Oct 15, 2008 at 3:36 PM | Permalink

      Re: Kenneth Fritsch (#529),
      Thank you for a thorough answer to my question.
      It appears that we have a serious case of “If you don’t get the result you want, adjust the data”. I believe that this comes from the same group that also continually ‘adjusts’ historical temperature data. Anything I can say about the apparent level of integrity in some of the work in climate science has already been said by others, so I will leave it at that.

DG

Pat Keating,
Could the referenced RSS adjustment be related to this?
ftp://ftp.ssmi.com/msu/readme_jan_2008.txt

Again however, there is still Randall & Herman
http://www.agu.org/pubs/crossref/2008/2007JD008864.shtml

also noted by RPS
http://climatesci.org/2008/01/01/important-new-paper-using-limited-time-period-trends-as-a-means-to-determine-attribution-of-discrepancies-in-microwave-sounding-unit-derived-tropospheric-temperature-time-by-rmrandall-and-bm-herman/