Comments on: Interaction of Infilling on Std Deviation

By: Steve McIntyre

Steve McIntyre — Thu, 19 Feb 2009 15:05:55 +0000

Tapio Schneider says above:

The initial variance estimate is biased because missing values are set to the mean (zero after centering), but the estimate is iteratively improved. (One can show that the estimate in the ordinary EM algorithm converges monotonically to the maximum likelihood estimate for normal data; in the regularized EM algorithm, this monotonic convergence is assured for a fixed regularization parameter but not necessarily in general, when regularization parameters are chosen adaptively.)

The following diagram of PCs is from the AVHRR data. Note the structure of the PC3. PCs recover patterns. The pattern being recovered in the PC3 is surely the infilling of data with zeros.

So notwithstanding Tapio’s comment, it sure looks like the infilling with zero pattern is preserved in the final result. As a caveat, while Steig said loudly that he used the unadorned Schneider algorithm, this statement contradicts a statement in the article that they used the algorithm as “adapted” by Mann and Rutherford. Previous adaptations by these authors have been criticized by Smerdon in three publications and it is possible that the phenomenon observed here is related to the adaptations rather than the Schneider algorithm itself. Without knowing what Steig really did – and he’s incommunicado – there’s still a lot of speculation here.

By: Sinan Unur

Sinan Unur — Wed, 18 Feb 2009 21:38:24 +0000

Re: conard (#89), the diff output contains < which is the start of an HTML tag. Unknown tags are ignored. So, it is best to replace all < signs with < before posting. -- Sinan

By: conard

conard — Wed, 18 Feb 2009 21:25:58 +0000

Re: conard (#88), Not sure what happened to the diff output but you can compare the years 1997,2008,2009 and see the small adjustments in the text files. The html files have another set of differences.

By: conard

conard — Wed, 18 Feb 2009 21:15:37 +0000

Re: Nic L (#86),

Excellent explanation. The station list for each give the exact same lat/lon/elevation. The data is slightly different between the AWS record and the Manned record, a minor curiosity.

$ diff Terra_Nova_Bay.txt Mario_Zucchelli.txt
1c1
Mario_Zucchelli temperature
12c12
1997 -0.8 -6.6 -15.0 -20.8 -20.8 -18.4 -23.8 -25.5 -19.5 -11.9 – -2.0
23,24c23,24
< 2008 – – -14.4 -16.5 -20.7 -17.1 -23.4 -21.9 -17.3 -16.1 -8.5 -1.1
2008 – – -14.4 -16.5 -20.7 -17.1 -23.3 -22.3 -17.3 -15.8 -8.5 -0.9
> 2009 -0.5 – – – – – – – – – – -

By: Ryan O

Ryan O — Wed, 18 Feb 2009 18:32:07 +0000

Re: Nic L (#86), Ah, makes sense.

By: Nic L

Nic L — Wed, 18 Feb 2009 18:31:06 +0000

Re: Ryan O (#80), MAario Zucchelli is the manned Italian base at Terra Nova Bay. It used to be called Terra Nova Bay but was renamed in memory of Mario Zucchelli after he died in 2004. My understanding is that the temperature data actually comes from an AWS named Eneide, ID 89262, located very near to Mario Zucchelli base (possibly within the confines thereof - access is listed as 5 minutes). It may be that BAS regard it as having the status of a manned station because of its location and ease of access - I'm not sure.

By: Nic L

Nic L — Wed, 18 Feb 2009 18:03:50 +0000

Re: Jeff C. (#77), I think we should be using the period 1957 to December 2006, but with the AWS data that Steig himself appears to have used. Most of the differences appear to be missing December 2002 data points in Steig's version, but there are a number of other missing data points, all as per my post on the Carnage thread. If anyone would like a copy of the AWS data that I believe Steig used, both in his original and in his corrected reconstruction, they are welcome to email me at erans99-2#AT#yahoo.co.uk and I will email a copy either in native R format or as a text file.

By: Sinan Unur

Sinan Unur — Wed, 18 Feb 2009 13:57:06 +0000

Re: MarkR (#83), what can I say, I agree. There, there is also the issue of adjustments ad nauseum as well as the fact that the number and composition of worldwide stations is changing. You might find the observations I made in http://www.unur.com/sinan/outbox/070816-us-is-only-two-percent.html interesting.

– Sinan

By: MarkR

MarkR — Wed, 18 Feb 2009 09:44:59 +0000

AnonyMoose, Sinan Unur. Doesn’t the whole infilling question also relate to the models, eg GISTEMP, which has wholesale (probably non random) infilling? Isn’t any Standard Deviation produced after infilling on such a scale worthless, and doesn’t that mean that any claims of “certanty” based on “infilled data” are not justifiable?

By: AnonyMoose

AnonyMoose — Wed, 18 Feb 2009 03:36:35 +0000

Re: MarkR (#50), Yes, I do suspect the standard deviation should be done before infilling. Perhaps there’s something clever being done in the rest of the process. If so, it will be interesting to see in what way this diluted standard deviation is relevant to anything.

Re: Ryan O (#80), If you haven’t noticed documentation that Terra Nova Bay is also Mario_Zucchelli then perhaps there is yet another BAS data problem. Their public change logging seems to be rather uncoordinated.

Site oddity: When I click “Submit Comment” I am told that I tried to open wp-comments-post.php, which is an MPEG4 video. That’s a mighty interesting confirmation technology you’re offering me. Should I perhaps be writing screenplays for replies?