http://www.ldeo.columbia.edu/~jsmerdon/2013_jclim_data/2013_jclim_smerdonetal.pdf

http://www.ldeo.columbia.edu/~jsmerdon/papers/2010b_jclim_smerdonetal.pdf

The book by C.R. Rao (Yes, THE Rao) and H. Toutenburg is an excellent buy if you want an uptodate account of the linear models. It is interesting to follow the work of the Team from this 400+ page book: Hegerl 2006 (total least squares, p. 70) is slightly more advanced (in terms of page numbers) than MBH (partial least squares, p. 65). Judging from that the next attempt will be minimax estimation (p. 72) or censored regression/LAD estimators (p. 80).

..and if I’m correctly linking this thread to Rao’s book, we are now on the Ch8, Analysis of Incomplete Data Sets . And the point I’m after is that do we have (8.4) Missing Data in the Response (solution ICE – type calibration ) , or (8.6) Missing Values in the X-matrix ( one-regressor solution, p. 257, looks like CCE – type to me ) . As I noted before, it seems to me that RegEM is close to ICE. I wrote short script to test this, here

And, as bender mentioned extrapolator vs. interpolator concept, you can try this script with missing values in the middle:

or missing values at the end:

It is quite clear that if values to be imputed values are not ‘like’ the calibration values, and SNR is as low as in all these proxy studies, regEM (tried without options! ) won’t work. CIs are based on calibration data, and that won’t do if we want to see whether current temperatures are unprecedented.

Kalman smooth CCE output, that’s my suggestion. Start with random walk model for temperature, it doesn’t have to be perfect model at the beginning

However, it still seems to me that synthesising data in an attempt to redress the lack of physical measurements in Antarctica is a worrysome procedure which would be difficult to justify to those policy-makers who currently wield absolute power in the world of climate science. Not that they any notice of anything that runs counter to their preconceived notions, of course.

Jeff C has regridded the Antarctic data according to proximity in an effort to reduce the dominance of high density stations over the rest of the reconstruction. Just by reasonable regridding (still not taking the proximity to the reconstructed station into account) the AWS trend is cut in half!!

http://noconsensus.wordpress.com/2009/02/15/aws-gridded-reconstruction/

Steve: this is good stuff. I’d done a thread linking to this post to attract more attention to your post.

The intuition behind is that we hope that the missing data
have little uncertainty given the incomplete data because the
EM algorithm implicitly assumes a strong relationship between
the missing data and the incomplete data.

http://www.cs.ucr.edu/~hli/paper/hli05rem.pdf I think we will get little traction from those using RegEM since this quote probably reflects what they do believe as scientists. However, a previous post had pointed out the problem with missing data not being random in many cases, but certain month(s). I see non-random data as a problem. Also, the spatial context must also have an assumed strong relationship for the paper’s temporal-spatial validity. Wouldn’t out of spatial samples (unused stations) be a way of testing this assumption of a strong relationship, and perhaps test the CI’s as well?

I’ll defer to Steve if he disagrees with this, but to me there is almost a philosophical event, a “paradigm shift” to use the language of 20 years ago.

Most science is directed to the discovery of new knowledge and its confirmation. Most science seizes upon observations that are different to the expected. Some branches of science love data that vary over large spans in complicated ways. But much of what climate science does is “homogenise”. Seek the lowest denominator, discard the interesting variations on the theme. Play a monotone instead of a melody.

Another paradigm shift. In the past, if mistakes were made and discovered, the authors and helpers would seek what went wrong and why. Often this was by breaking up the bigger problem into bite-sized pieces, controlling as many variables as possible, then saying Eureka when a subset revealed the error. Climate people on the other hand commonly deny error and react by contriving hideously complicated models that have little chance of finding fundamental errors because they are poorly formulated originally.

When PCs were coming available for my geological fratenity in the early 80s, we encouraged data gatherers to still plot maps by hand, the old way, rather than just plug the numbers ito a package. Numbers take on a personality when you worry them enough. That can help you do a better job. But, to lump a huge mass into an amorphous machine and place your faith in a one-page printout – that is not science, that is becoming a servant of the machine.

It is for reasons like this that I dislike the brute force solution to the 4 colour map problem. It’s not really proven, it’s just shown very unlikely to be wrong.

Forget the global average, handle sub-sets with which you can become intimate and start to understand them.

When you understand a few, then aggregate your knowledge. And again and again.

Exactly. Then you won’t use nonsense data like the Finnish sediment data that showed when the bridge was built. But my other point was that the problems start with the selection of a statistical model. Once you decide to use RegEM, or lots of similar methods, as I understand it you are pretty much obliged to infill. But if you used simpler methods you’re not. If you analyse trends at individual locations missing data is not a problem. But if you were tempted to infill missing data at a single location and use data from other locations it would be pretty obvious that what you were doing was flawed. And if you felt you had to infill it would be obvious that the data at that location was inadequate.