Fair enough all the way around Ryan. You’ve certainly done more than your fair share of work here. It’s just become a pet peeve of mine whenever people use the cost of hard disk space (not that you were doing it here, just suggesting that others might) as an excuse for not keeping data.

I realize it’s probably not even Joey’s fault, but rather the NASA data center types, who display the famous MASH supply depot sergeant mentality: “I’ve got three incubators I don’t need. But if I gave you one, then I’d only have two, and two is not as good as three.”

Enough of my ranting. Carry on. Sorry.

Good grief. Maybe I’m just cranky lately because April has been so darn cold her in the Midwest.

But a 2 terabyte hard drive can be had for about $300 these days. Maybe we should contribute to the tip jar to buy Joey one.

Ryan,

The original data is a huge huge data set, and I don’t actually have the
original data myself. Co-author Joey Comiso is working to put all those
data on line at NASA, but is still working on securing the server space
for it.

In any case, 1981-2005 ought to be sufficient for reproducing our
results, if that’s what you’re interested in looking at.

Eric

.
In this case, I fully understand why Steig would not, himself, have the original data (it’s about 1.2 terabytes big). So no big deal there. I’m hopeful that I will receive a response from Comiso, since I had copied him on the request as well. Anyway, this is what I sent back to Steig:
.

Dr. Steig,

Thank you for the quick response. And yes, the original data is quite massive. As you personally do not have the data, if it is acceptable to you, I will defer further questions about the remainder of the data to Dr. Comiso.

Again, thank you for the reply.

Best regards,
Ryan

.
Hopefully Comiso really is in the process of finding server space for the data. I do know that NSIDC doesn’t presently have enough space, which is why they had to collate it for me in bunches (2-3 years at a time), host it for a few days to allow me to download it, then delete it and replace it with another batch.

Dr. Steig,

I have a request concerning your 2009 paper in Nature in which you present a reconstruction of Antarctic temperatures from 1957 to December, 2006. On your webpage (http://faculty.washington.edu/steig/nature09data/), you provide the cloudmasked AVHRR data set. You also provide the link to NSIDC for obtaining the raw AVHRR data. However, NSIDC does not have AVHRR data archived past 2005.

NSIDC has been kind enough to supply me with the 5-km gridded data from 1981-2005, but they were not able to supply the remainder of the data as used in your paper. Because I am unaware of any public source for the remainder of the data, I respectfully request that you supply the missing data, or, if a public source exists, that you direct me to that source. Your assistance is greatly appreciated.

Best regards,
Ryan O’Donnell

.
I’ll let you guys know how it turns out.

What CASPR Does Not Do
Temperature and humidity profiles are not retrieved, they are input.
Calibration of AVHRR raw data.
Navigation and registration of raw data are not performed.
Retrievals of some parameters for large solar zenith angles.
Retrievals outside of the polar regions, although the primary limitation is the surface temperature retrieval, which could easily be expanded to include lower latitude oceans and land.
Stratospheric clouds are not retrieved; all clouds are restricted to the troposphere.

CASPR is research code. It does a lot of things well but doesn’t do anything perfectly. Many of the algorithms have been validated, some have not. They are all detailed in the Reference Guide. There are three broad problems worth mentioning at the outset. First, everything depends on cloud detection, which sometimes borders on being as much an art as a science when working in the polar regions with the AVHRR. We are, indeed, trying to squeeze water from a stone. We do not claim to have solved the cloud detection problem, but rather provide methods that work reasonably well most of the time.

Third, the retrieval of cloudy sky parameters requires temperature and reflectance values underneath the clouds. CASPR interpolates clear sky values to cloudy areas. This generally works but can result in large uncertainties in very cloudy areas. Cautionary notes are given throughout the Reference Guide. Please do not ignore them!

This answers some of my initial queries about how satellite derives surface temperatures are measured under cloud. They are not. Thus, the reconstruction of a temperature map of the Antarctic at a given time includes factors that are stated to be unrelaible, yet we end up with claims of 0.1 degrees C discrimination.

I am unsure of the measurements that go into the statement I bolded “This generally works, but …..”

Maybe there has been an advance that I am unaware of, but to me it seems that a good guess (or a bad one) has made the cover of “Nature”.

When you performed your calibration, did you apply the same correction to each station within each of the five groups? I got that impression from reading the text but wanted to make sure. I think you need to do it that way as opposed to each station getting its own tweak.

To me, it looks like the geographical breakdown is something like this:
East Coast/Peninsula (red)
East and West Interior (black)
Ice shelves (blue)
Ross Sea Coast (green)
West coast (light blue) – this one is iffy, but there aren’t many points

These breakdowns make sense to me. It is not so much the region as the commonality of the physical environment.

One last point and this is totally off the wall. I realize they are completely different things, but did you notice how much your Wilcoxon test plot looks like the MSU temp anomaly plot? Coincidence? Probably, but they are quite similar. In fact, when I was just scanning the post I thought that was what it was.

Ryan, very impressive post. Thanks for all the work. Regarding NOAA 11, is it possible that Pinatubo interfered with the signal recognition at the tail end of this period?

The first thing to note when plotting the AVHRR data against the ground temperatures (all manned and AWS stations) is that it appears to contain multiple populations that are not all equally correlated to ground temperatures. This could cause several problems when trying to determine satellite offsets, such as:
1. The multiple populations increase the data scatter, which decreases the ability to identify offsets.
2. A poorly correlated population with data concentrated only in certain times could cause mistaken identification of an offset.
3. Some of the populations are not related linearly with ground temperatures, which would exaggerate or suppress the magnitude of a calculated offset.
So the first thing we would need to do is identify the populations. This proved to be a somewhat challenging proposition, as they are all intermixed in the higher temperature range. After much trial and error organizing groups, plotting, reorganizing groups, replotting, etc., five distinct groups emerged:

In the R script, I retained and documented the plotting functions I used to help do the grouping. Function plt.stn() allows you to plot a particular station vs. the groups. If you want, you can go through it just to verify that there are, indeed, five separate groups and that I have the correct stations assigned.
After identifying the groups, we should check to see if there may be some physical reason that the AVHRR temperatures at different station locations would behave differently. The first thing would be to look for geographical significance (NOTE: The colors DO NOT match the above plot).

The main group, which was the long, skinny group in red on the scatter plot, corresponds to the Antarctic interior. The other groups are coastal. If I had to venture a guess, I would say that the difference in the shape of the curves is due to reflectivity differences between water, snow (which also changes with grain size), and ice. If so, this effect was also described (for 37GHz measurements) in Shuman (2001) linked by Roman earlier:
http://ams.allenpress.com/archive/1520-0442/14/9/pdf/i1520-0442-14-9-1977.pdf
Now that we’ve identified our groups, we need to calibrate them to the ground temperatures. Doing this on a station-by-station basis would be suspect since many of the stations have a small number of points. Within a group, however, we have a much larger number of points, so we can be more certain of our transforms.
The process of doing the calibration (after trying lots of things) ended up being fairly simple:
1. Bias correction
2. Nonlinearity correction
3. Fine bias correction
4. Fine nonlinearity correction
The result:

The next step is to convert to anomalies. Care has to be taken here. Remember that the purpose is to try to determine offsets between satellites. Because the ground data is discontinuous with large chunks missing, simply making the base periods the same when converting to anomalies is not enough. Instead, we will convert to anomalies using the entire time frame (1982-2006) and using ONLY months for which there is corresponding ground data. This makes sure that the comparison between the calibrated anomalies and the ground anomalies is an apples-to-apples comparison.
After converting to anomalies, we need to find some test to determine if there are statistically significant offsets between the satellites. For this we will use a paired Wilcoxon test (since the residuals are non-normal – I checked) with a 24-month range. The estimate of the difference in means will be normalized to the 95% confidence interval to allow continuous plotting of the points as we move through all 300 rows of the data sets. If there is a statistically significant offset between satellites, we will see a peak, approximately in the center of the satellite coverage period, that exceeds 1.0:

The biggest feature is the huge spike with NOAA-14. Without a doubt, there is a statistically significant offset with NOAA-14. NOAA-7 and -9 are also low; NOAA-11 looks generally okay except for the massive dip at the end (which I have not come up with a satisfactory way of handling yet); and NOAA-16 and 17 also look okay.
Now that we’ve convinced ourselves that the offsets are real, it is time to calculate them. We obtain:

-0.136315035 -0.217185496 -0.097247497 0.215448620 -0.006319678 -0.195520508

It’s pretty obvious that these factors will reduce the trends. We get a continent-wide trend of 0.074 +/- 0.158 (compared to 0.187 +/- 0.151 from the Comiso data).
However, had we simply calculated offsets without going through the above calibration, we would have gotten:

-0.14944679 -0.16962065 -0.04562425 0.33161241 0.09104473 -0.05796546

Note that this would have even further decreased the trends – to the tune of a continent-wide average of 0.032 +/- 0.149.
Original Comiso trends (deg C/decade and 95% CI):

Peninsula 0.406552585 0.1925186
West Antarctica 0.411971312 0.2217650
Ross Ice Shelf -0.104963672 0.2206460
East Antarctica 0.225650354 0.1942384
All 0.187422507 0.1510635

Calibrated trends (deg C/decade and 95% CI):

Peninsula 0.29165083 0.2282490
West Antarctica 0.26177232 0.2502632
Ross Ice Shelf -0.12965218 0.2714344
East Antarctica 0.06001991 0.2033417
All 0.07399090 0.1572431

Here’s a plot showing my geographical groupings:

Teaser:
Hm.
Trends about halved – very similar to what the Jeff’s got by regridding. Common theme, maybe? Suspiciouser and suspiciouser . . . but that’s enough for now. There’s a lot more in the script I posted – you can compare the main, PCA, and AWS recons as well. There’s also some single value decomposition at the end which isn’t finished yet and will be the subject of another post. Until next time, however, I will leave you with this curious plot:

The blue line is simply a slope of 1, provided for scale.
Unlabeled, one might have mistaken this for the Small Magellanic Cloud: