http://noconsensus.wordpress.com/2009/07/06/tav-to-realclimate-you-can't-get-there-from-here/

Iterative and eigenvector weighting show substantial improvements in stability and accurately imputing withheld data.

My thanks as well. It’s fantastic to have the authors of these papers stop by from time to time. From some of the work done on CA, I think the result from your method will be quite almost the same as RegEM on most datasets with substantially less computing time.

I just came across your discussions on our paper

http://hdl.handle.net/2268/4291

and would like to take the opportunity to respond to some of your questions/comments.

Our method always assumes that clouds are correctly masked (so whether clouds are colder or warmer is a question for the groups that provide the masks). Systematic errors here will always affect the reconstruction and probably lead to cold bias for cold clouds incorrectly masked. To help the user, we are currently testing a posteriori quality control for each pixel and firsts results are promising in the sense that undetected haze and cloud edges are automatically flagged and can be masked for a second filling iteration.

Whether our method is more “true” than the RegEM approach cannot be proven in general but can be tested in each case, for example setting aside some data under artificial clouds and then calculating the reconstruction errors for both methods there (as suggested by Nicholas). Hopefully both methods provide similar results, otherwise it means that there are not enough data constraints available to fill in clouds. In fact this artificial cloud approach is used in our method to optimise the number of PC to retain and provides also a global error estimate for the reconstruction.

For the regions with almost permanent cloud coverage, our method will never provide a reasonable reconstruction, because there is no a priori spatial covariance function defined. Hence the method has to derive the spatial covariance from the data themselves, which is not possible if there are insufficient data in one location. This is why we apply a “land” mask also in points with a too large cloud coverage in time. In other words, the filling can only be done in places where there is sufficiently statistical evidence that allows to infer values there knowing observations in other locations and other moments.

Since the 2003 paper we added a fundamental tool to the reconstruction itself, the associated error field. There interesting point here is that, as expected, under clouds the errors are larger than elsewhere, but depending on the PC structures, some locations are better constrained than others for the filling.

Specially the error fields are of importance when dealing with the computation of spatial averages for detecting trends, as the imputation error influences the mean values and the standard error on the mean.
See for example our discussion in http://www.ocean-sci.net/2/183/2006/os-2-183-2006.html which seems important for the case you discuss here; ie detecting weak temporal trends in spatial averages, themselves depending on errors in the interpolation. The most critical part here is that errors might be spatially correlated which has a strong influence on the error on the spatial average and hence the confidence you can have on the trends.

For further understanding and reading on our method, from the original 2003 version it was later optimised for efficient use of Arpack library

http://hdl.handle.net/2268/4296

and later extended with error estimates

http://www.ocean-sci.net/2/183/2006/os-2-183-2006.pdf

and multivariate approaches

http://hdl.handle.net/2268/9485

The code itself and documentation is available

http://ocgmod2.marine.usf.edu/DINEOF-welcome.html

http://ocgmod2.marine.usf.edu/mediawiki/index.php/DINEOF-_User_guide

Also do not hesitate to contact us for the questions and possibly a comparison with other methods.

The results show that the cloud-free only monthly average is colder than the true monthly average by about 0.3°C with a standard deviation of about 0.6°C during summer and 0.5°C with a standard deviation of 1.5°C during the winter.

You probably are going to obtain more information by doing the station by station differences. The information provided from Cosimo above is not sufficient, for example, to determine a standard error – without some big assumptions.