RE: “Response: At this point, forecasting hurricane trends based on the supposed future of the AMO seems more speculative and less based on fundamental physics than factoring in the contribution of anthropogenic global warming. –raypierre”

So, are you implying some sort of conspiracy, where NOAA are silencing those who share your own bias, and are promoting those who are, as you would call them, “deniers?” You actually believe that NOAA would risk poor forecasts to satisfy some sort of “Bushie political agenda?” You actually believe that? And you call yourself a scientist? Got tin foil? ….

by Steve Sadlov

“After reading several of the comments, I had to chime in given that I formerly studied to be both a statistician and atmospheric scientist but am now engaged in creating highly sophisticated economic/statistical models for one of the top 5 largest hedge funds. Because of the proprietary nature of what I do, I cannot go into the nature of the models that I create and work on, but suffice it to say that both the complexity and computing power required are entirely on order of many GCMs – one may notice how many big/fast computers in the top 500 list are at financial institutions or trading organizations.

Modeling financial markets is brutally difficult. The fundamental relationships are highly nonlinear and the associated data possess pathological distributions. Even worse, the nature of the markets means that one does not have a convenient set of Navier-Stokes equations upon which to ground the modeling, but instead has to deal with:

1) Underlying equations that may not be knowable from any form of first principles, and
2) Even when known, equations that are nonstationary in a very nasty and nonlinear way owing to the psychological/sociological aspects of financial markets

More importantly, we have a very natural measure of skill. It is:

1) Be better on a relative level than our competition, and
2) Be right significantly more often than we are wrong

The climate community too often embraces only the first aspect. On the other hand, if we ignore either one of our “skill” dimensions in my line of work, then the lights get turned off and we are out of business.

Moreover, because of the very natural feedback of our industry, we cannot “game” our models. For example, I cannot create a model that looks great in testing but loses money in actual use; the finance literature is full of such models, but using them will cause one to go bankrupt. In the same fashion, I cannot examine behavior in individual financial markets and then “tweak” various differential equations and statistical parameters so that the model forecasts better match that market’s returns – such gamed models also fail miserably when used in reality.

Now I will tie this in more directly with climate research. The measure of skill quoted by Pielke from Annan’s website:

“Forecast skill is generally defined as the performance of particular forecast system in comparison to some other reference technique.”

is exactly the definition I would expect from someone who does not have to be right more often than wrong. In fact, unlike what I do in the applied finance world, there has been little feedback whatsoever on whether or not climate model forecasts created to date have been correct thus far. Instead, the feedback seems to be how well one can fit a GCM to an observed data set in a publication (such data include historical near-surface temps, satellite observations, etc.) to see if it indeed matches historical data. Even though such GCM forecasts are integrated over time, the continual process of re-running and re-running them until better matches are created is a form of in-sample maximization that provides no information on what the model’s realized forecast efficacy will be (even the in-sample matches are currently poor, as has been noted above).

From an outsider’s viewpoint, I have to ask a troubling question: What are the incentives to create an accurate climate model forecast going forward? I am not talking about short-term (upcoming season) or even early medium-term forecasting, but the long-run enhanced CO2 emissions GCM forecasts. Who will validate these forecasts? Will the validations be during the researchers’ careers? One can cynically note that there is certainly incentive to create attention-getting forecasts, a result that is generating a lot of climate research funding without any direct tie-in as to whether or not the forecasts will later prove to be true (which, I should note, they very well could be right after all).

I am left with only two incentives to get the forecasts right: The integrity of the researchers/groups, and the overall integrity of the scientific process. I do believe that most researchers have such integrity, including those posting here, but I call into question the scientific process here, especially in the short-run. “Group think” behavior has been a stunning and recurrent problem throughout the history of scientific progress. And it can flourish when there is a lack of experimental verification and predictive accuracy feedback, especially when performance measures do not require one to be right. I know that in my world, if we could be successful under such lax performance measures, my own life would be a lot less stressful. “

But Pielke is not proposing that the computer models should “do better at matching the actual climate than the measurements of climate themselves” as # 43 proposes. For example, Pielke says:

Skill can be quantified, for instance, by a model forecast as having a root mean squared error of 1.2C with respect to 500 hPa temperatures, for example. This is one type of model “skill”.

The idea that Pielke is asking that the computer models should do better than the observations is a straw man. Pielke is only (and reasonably, in my opinion) asking that the results be what I call “lifelike” — that the standard deviation, mean, first derivatives, etc. of the model results be similar (in some specified mathematical sense) to the observational data.

The appropriate metric is not, as Annan proposes, whether a computer model does better than some alternative method such as persistence or climatology. It is whether the results are in close agreement (again in some specified mathematical sense) with observational data.

Annan claims that

There are no examples using your definition to be found in the literature, because it simply doesn’t make sense.

This is simply not true. The “Gerrity Score”, for example, is used by the UK Met Office to assess the “skill” of their forecasts. From their web site:

The Monthly Outlook includes forecasts of expected temperature and rainfall categories. Five categories are used; (1) well below average, (2) below average, (3) near average, (4) above average or (5) well above average conditions for the time of year.

To assess the accuracy of the forecast we compare the predicted category with the category that was actually observed to occur. We use a points-based scoring system in which maximum points are awarded to forecasts that are ‘spot on’ (i.e. the forecast category exactly matches the category that actually occurred), fewer points are awarded for ‘near misses’ (e.g. the forecast is wrong by one category), and points are subtracted for misleading forecasts (i.e. a forecast of above normal when below normal is observed). The score used is called the Gerrity Skill Score (GSS), and is one of the scores recommended by the World Meteorological Organization (WMO) for evaluation of long-range forecasts. The score is designed so that forecasts that are always ‘spot-on’ would achieve a score of 1.0, and forecasts based on simply ‘forecasting’ the long-term average (category 3) would receive a score of zero. Thus a positive score means the forecast is better than guesswork and better than assuming future conditions will be similar to the long-term average. Although the theoretical maximum score is 1.0, best scores achieved at the monthly range are of order 0.6, and found in the more predictable tropical regions.

Note that they are not comparing to other forecasts, but to observational reality.

I have repeatedly asked Gavin Schmidt whether they use the Gerrity Score or some other method to assess the “skill” of their models. To date, he has refused to reply to the question.

w.