Re: Steve McIntyre (#47),

The Jeffs are experimenting with the algorithm under the assumption that Steig’s methodological description is untrue

Not sure I’d call it untrue, just deliberately misleading (not that that is any better). In the methods section Steig has the following sentence.

We use an adaptation of RegEM in which only a small number, k, of significant eigenvectors are used

I think adaptation is the key word in that sentence. His adaptation is not running RegEM with regpar=3, as that is a standard use. His adaptation is using three eigenvectors as the input to RegEM. He could have just come out and said that, but the sentence above is a work of art. It obfuscates, but at the same time allows the claim that the method was revealed.

Re: Layman Lurker (#46), Regarding the whole cloud mask issue there are three things I have found interesting from studying the UWisc data set.

1) There is a clear discontinuity in cloud cover between NOAA-14 and NOAA-16 (Dec 2000/Jan 2001). I have traded emails with those responsible for the dataset at UWisc and they were very candid in stating that the NOAA-16 data has problems and they don’t know how to fix it. I wonder how Comiso dealt with this.

2) The average continental cloud cover ranges from roughly 45 to 80% depending upon the season. West Antarctica and the East Antarctica coast are particularly prone to clouds, more than 75% of the days would be thrown out as cloud covered. How does Comiso deal with this when calculating monthly means? Does he simply calculate a mean using the remaining days of data or is some sort of infilling algorithm used?

3) The areas Comiso shows warming appear to correlate with areas of heavy cloud cover. This could be entirely spurious, but it is interesting.

Re: Hu McCulloch (#44), Regarding the +/- 10 deg C climatological mean filter, this also would appear to cause a large percentage of the remaining days to be thrown out. The UWisc dataset has a fair number of *months* that exceed +/- 10 deg C. Think how many days must exceed the threshold to have months exceed it. I’m trying to get a better handle on how many days that might include by doing some statistical analysis on the monthly data. My initial estimate is that a 10 deg C *daily* filter would be roughly equivalent to a 5.8 deg C *monthly* filter. It is still very preliminary, but it means we could be down to only 10 – 20% of the daily data being used to calculate monthly averages for wide areas of the continent.

If Comiso was listening, I’d request his cloud mask files and his final masked daily data prior to calculating monthly means. I’d like to see just how many actual days of data were left after all the masking and filtering. I’d also like to know how he worked around the data contamination and diurnal drift issues that appear to affect NOAA-16.

I doubt I’ll ever see it. They released the satellite data (albeit a filtered and infilled version), I expect any further requests will be seen as unreasonable. Jousting with jesters, so to speak.

I tried running the script and it hicpped on me in midstream ( – could have benn my fault). Anyway, I did some reading up on the methods while I was working on the stuff and it helped me to get some insight on the methods.

At the same time, I found some ways to make the program more efficient. Wrote a script to identify and group similar patterns of missing values cutting down the number of times ttls needs to be called.

Your comment prompted me to go back and finish a replication of the satellite surface station reconstruction.

http://noconsensus.wordpress.com/2009/04/05/steig-avhrr-reconstruction-from-satellite-data/

Roman, don’t forget that I’ve done an R emulation that is reconciled to the Jeffs Matlab runs. It was a lot of work. ):

http://data.climateaudit.org/scripts/regem/regem.functions.txt

Dunno. I haven’t personally had any access to NSIDC data which requires an application protocol. Given that Ryan O has done so, I’m waiting to see how he does before thinking about it further.

We’re also a bit stuck on the next step. The Jeffs are experimenting with the algorithm under the assumption that Steig’s methodological description is untrue – i.e. that they did not, as claimed, apply RegEM directly to the AVHRR+surf data, but reduced the AVHRR data through some undisclosed method, perhaps PC. This is speculation on their part, pure and simple. I’m following their experiments, but it would definitely make sense to ask Nature to require PR Challenger Steig to provide further particulars on this step at some point fairly soon.

Without data, this question is pretty much a dead end.

You are probably right. However hopefully it can at least be narrowed down further. Jeff C. is exploring the differences between NSIDC AVHRR and Steig AVHRR – with particular interest in those grids corresponding with surface stations. Hoping to identify where and when cloud masking was done and what the masked values were. Also looking for any clues for possible role of surface stations in calibration for infilling. Ryan O. has also been studying cloud masking and says he hopes to post on calibration of infilling as well.

Re: Hu McCulloch (#44),

Then, does 10dC clipping plus straightforward time and spatial aggregation essentially yield Steig’s cloudmaskedAVHRR.txt file, or are there other protocols and intermediate data sets we should be insisting on?

Dunno. Hopefully, when Jeff C. and/or Ryan O. posts on this, we will see if it is as simple as adding the +/- 10C rule to the NSIDC AVHRR data.

My taxes aren’t due until the end of April so I have been ignoring them.

I have been trying to cobble up my own R script for TTLS and RegEM from first principles but I haven’t quite got it right yet. It has lead me to notice that the truncation to relatively few principal components seems to amplify those series which are the most variable and suppress series which have less fluctuation in them. This effect is not a result of any trends in the various series since re-ordering the rows of the data matrices (i.e. shuffling the times of all of the series) should have no calculational effect on the values of the PCs. Choosing too few PCs can lead toreconstructions with trends similar to those exhibited by the more variable series.

With regard to Steig, I have no data that I would ask for at this time, but I do have some questions as to the specific way his methods were applied. For example, the AVHRR data scraped by Ryan and the Jeffs had missing values including several missing months. However, the version posted on the web site (which was claimed to be the data fed into the Regem procedure) was complete. Exactly how was the infilling done? An alternative approach could have been to relegate the missing months to be reconstructed by Regem at the same time as the pre-1982 reconstruction was being done. This apparently was not what they did.

When the cloud masking adjustments were done, was the extreme data (outside of +/- 10) merely cut back to these limits or was it removed completely and infilled along with the missing months?

Were three PCs extracted from the latest posted data and used in Regem or was something different done and then a second unmentioned procedure applied to mke the final reconstruction decomposable into 3 pieces? It seems that if it was the former, we would have already seen a duplication of their results (or have I missed something?). This would assume that we know exactly what the station data they used looked like and how or when any infills may have been applied to that as well.

I am sure I can think of more questions, but that would do for now.

The Comiso AVHRR “raw” data is net of cloud adjustments but there’s no data on Comiso clouds matching the UWisc data. Without data, this question is pretty much a dead end.

Nature’s policy still requires Steig to provide all the data and “protocols” necessary to replicate his study. Which data precisely is missing? CA contributers have been slowly downloading gigatons of data from NSIDC, but is this complete, or does it only run up to 2004 as per the NSIDC webpage? Did he somehow get newer data that is not yet available from NSIDC?

Then, does 10dC clipping plus straightforward time and spatial aggregation essentially yield Steig’s cloudmaskedAVHRR.txt file, or are there other protocols and intermediate data sets we should be insisting on? Are there some months and 50km regions with no after-masking data at all that must have been filled in by an as yet undisclosed protocol? Or is it obvious how to do this?

yep, for example, in the UWisc data considered by Jeff C, the relations between cloud and satellite life cycle were extraordinary. The Comiso AVHRR “raw” data is net of cloud adjustments but there’s no data on Comiso clouds matching the UWisc data. Without data, this question is pretty much a dead end. Comiso’s adjustments for this study are different than UWisc; blanket statements at RC that the “code is available” are untrue.