Comments on: How much Estimation is too much Estimation?

By: The Accidental Tourist « Climate Audit

The Accidental Tourist « Climate Audit — Thu, 14 Jul 2011 02:39:30 +0000

[...] I viewed the data using both a 1200km and a 250km smoothing radius. Doing so helped me gauge the station coverage and the extent the 1200km smoothing algorithm estimates temperatures over the vast unsampled swaths [...]

By: Is It Getting Warmer? » Are the numbers real?

Is It Getting Warmer? » Are the numbers real? — Tue, 12 Aug 2008 16:36:28 +0000

[...] seen is written by Steve McIntyre over at Climate Audit in two separate discussions. From February 9th: Unfortunately, the previous year’s December temperature is not included in the first year of [...]

By: Watts Up With That?

Watts Up With That? — Tue, 12 Aug 2008 01:31:15 +0000

[...] this year I did a post on the amount of estimation done to the GHCN temperature record by GISS before generating zonal and [...]

By: wkkruse

wkkruse — Sat, 16 Feb 2008 14:34:03 +0000

John Goetz #95
My take on this now ( and I assume it’s the same as your take) is that the GISS process can be thought of as estimating the deviation of the temperature in month a from its long term average. And it does this as as the average of the deviation of months b and c from their respective long term averages. In other words,
DEVa=[(Tb-TB) +(Tc-TC)]/2. Then the estimated temperature for month a is TA+DEVa which leads to your result for Tq.

Thinking about it like this makes it sound more reasonable. I would hope, however, that GISS tested this method against other alternatives.

I’m going back to lurking.

By: John Goetz

John Goetz — Sat, 16 Feb 2008 13:51:08 +0000

#93 wkkruse

It should be deltaTn = Tn – TN. When I was solving the problem back in September, I figured Hansen was adding a bias to the month’s average in order to get the month’s estimate. So my second step in that derivation is correct. I got sloppy with the Latex in the fourth step, which should read deltaTn = Tn – TN instead of deltaTn = TN – Tn. The signs are backwards in the fifth step as well. Fortunately, I had worked it out in a notebook and was trying to copy from my notebook into the Latex, and I did copy the sixth and final step correctly.

When I get a chance tonight I will post a complete and corrected derivation – hopefully without the typos. What I posted in the past is incomplete in that it does not specify when rounding of results occurs, nor does it specify when the seasonal averages are calculated.

By: wkkruse

wkkruse — Sat, 16 Feb 2008 12:46:16 +0000

There is a typo in #92. It should say that the final result Tq=TA/3+(TB+TC)/6+(Tb+Tc)/6

By: wkkruse

wkkruse — Sat, 16 Feb 2008 11:41:10 +0000

John Goetz #91
In your post of Sept 6 to which you referred me, the next to last line in your derivation of Tq is
Ta=TA+(TB+TC)/2 -(Tb+Tc)/2 where Ta is the estimated temperature for mont a that is missing data. (This comes about from deltaTn=TN-Tn) Since Tq= (Ta+Tb+Tc)/3 then by substitution
Tq=[TA+(TB+TC)/2 -(Tb+Tc)/2 +Tb+Tc]/3 = [TA+(TB+TC)/2+ (Tb+Tc)/2]/3=TA/3+(Tb+TC)/6+(Tb+Tc)/6.

On the other hand if deltaTn=Tn-TN then I get your result for Tq.

By: John Goetz

John Goetz — Sat, 16 Feb 2008 03:56:47 +0000

#90 wkkruse

Thank you for pointing out the error in comment #76. It should read Tq= rather than Ta=.

I do not get your resulting equation, however. I would need to see your step-by-step derivation to comment on it. While I now see that I had an error in the intermediate result in the post I referred you to, the resulting equation is still correct (as far as I can tell). I’ve rerun it a bunch of times on GISS data and I get the same results as Hansen.

One trap to watch out for is that the December value you must use is from the previous year, not the current year. That is a programming error I recall stumbling into.

By: wkkruse

wkkruse — Sat, 16 Feb 2008 00:59:20 +0000

John Goetz #89
In # 76 above you said that Ta was the temperature of the missing month. I looked at your earlier derivation cited in your reply in #89, and it seems that your equation for Ta in # 76 is supposed to be the estimate for the quarter that is missing the temperature in month a. However, in reviewing your earlier derivation for the quarterly estimate, I can’t get your result. I get (using the notation in #76 above) that
Tq=TA/3+(Tb+Tc)/6+(TB+TC)/6, which I think makes more sense since it properly weighs the 3 months.

By: John Goetz

John Goetz — Fri, 15 Feb 2008 23:01:22 +0000

#86 wkkruse

Actually, the equations are mine. I have never seen the equations as I have described in Hansen’s code, nor have I seen them in his written material. We had discovered that Hansen estimates a missing monthly temperature by finding the average temperature for that month and adding to it a delta temperature. The delta temperature used is the average of the delta temperatures for the other two months in the season. I derived the final equations as shown above so that I would have one simple expression to program. Through a few trials I also figured out when he does his rounding. See here for how the equations were derived.

#84 Sam

Yes, I have played around a lot randomly removing months from the record and seeing how close I get. I can’t say I see any particular pattern or trend in the results, other than I have yet to see it produce the actual temperature. I’ve toyed with doing a Monte-Carlo style simulation where I make hundreds or thousands of runs against the same set of stations, randomly removing records and seeing what kind of distribution I get out of it. Right now it is just a thought.

#81 Paul

Yes, a month with unseasonable weather is generally washed from the record using this method. While no estimation method is perfect, I do have have two problems with what is being done here.

First, the method used has the potential to change old estimates every time new data is added to the record. I’ve run a number of experiments and have seen that happen very, very frequently. The changes may not be huge (0.1 to 0.8 degrees perhaps) and the estimates seem to converge on a single value after a while (although no one knows how accurate that value is), but it bothers me to know that from one year to the next the old data is changing.

Second, I see no reason to estimate monthly or seasonal values at all. Hansen chooses to run his gridcell analysis using annual temperatures, but why he can’t he just do it with the available monthly temperatures at any given point in time? Then, at the end of the year take the twelve monthly averages and calculate an annual average. We could argue that different stations contribute to the monthly average, but that happens with the annual temperatures anyway.