Following excellent notes transferred from a comment by Ryan O, who has been doing excellent analyses of the Steigian swamp.
As I had mentioned before, there appear to be unaccounted-for offsets between the different satellites that make up the Comiso AVHRR cloudmasked data. I have spent a while trying to determine first if the offsets actually exist; and, second what the result of correcting for them would be. The R script and two supplemental data files you will need to be able to replicate this are:
R Script: http://www.mediafire.com/download.php?dtuntzxadzz
Station Information: http://www.mediafire.com/download.php?ioyiizcmmzn
Updated READER Temps: http://www.mediafire.com/download.php?mygwgedfa4z
The first thing to note when plotting the AVHRR data against the ground temperatures (all manned and AWS stations) is that it appears to contain multiple populations that are not all equally correlated to ground temperatures. This could cause several problems when trying to determine satellite offsets, such as:
1. The multiple populations increase the data scatter, which decreases the ability to identify offsets.
2. A poorly correlated population with data concentrated only in certain times could cause mistaken identification of an offset.
3. Some of the populations are not related linearly with ground temperatures, which would exaggerate or suppress the magnitude of a calculated offset.
So the first thing we would need to do is identify the populations. This proved to be a somewhat challenging proposition, as they are all intermixed in the higher temperature range. After much trial and error organizing groups, plotting, reorganizing groups, replotting, etc., five distinct groups emerged:
In the R script, I retained and documented the plotting functions I used to help do the grouping. Function plt.stn() allows you to plot a particular station vs. the groups. If you want, you can go through it just to verify that there are, indeed, five separate groups and that I have the correct stations assigned.
After identifying the groups, we should check to see if there may be some physical reason that the AVHRR temperatures at different station locations would behave differently. The first thing would be to look for geographical significance (NOTE: The colors DO NOT match the above plot).
The main group, which was the long, skinny group in red on the scatter plot, corresponds to the Antarctic interior. The other groups are coastal. If I had to venture a guess, I would say that the difference in the shape of the curves is due to reflectivity differences between water, snow (which also changes with grain size), and ice. If so, this effect was also described (for 37GHz measurements) in Shuman (2001) linked by Roman earlier:
Now that we’ve identified our groups, we need to calibrate them to the ground temperatures. Doing this on a station-by-station basis would be suspect since many of the stations have a small number of points. Within a group, however, we have a much larger number of points, so we can be more certain of our transforms.
The process of doing the calibration (after trying lots of things) ended up being fairly simple:
1. Bias correction
2. Nonlinearity correction
3. Fine bias correction
4. Fine nonlinearity correction
The next step is to convert to anomalies. Care has to be taken here. Remember that the purpose is to try to determine offsets between satellites. Because the ground data is discontinuous with large chunks missing, simply making the base periods the same when converting to anomalies is not enough. Instead, we will convert to anomalies using the entire time frame (1982-2006) and using ONLY months for which there is corresponding ground data. This makes sure that the comparison between the calibrated anomalies and the ground anomalies is an apples-to-apples comparison.
After converting to anomalies, we need to find some test to determine if there are statistically significant offsets between the satellites. For this we will use a paired Wilcoxon test (since the residuals are non-normal – I checked) with a 24-month range. The estimate of the difference in means will be normalized to the 95% confidence interval to allow continuous plotting of the points as we move through all 300 rows of the data sets. If there is a statistically significant offset between satellites, we will see a peak, approximately in the center of the satellite coverage period, that exceeds 1.0:
The biggest feature is the huge spike with NOAA-14. Without a doubt, there is a statistically significant offset with NOAA-14. NOAA-7 and -9 are also low; NOAA-11 looks generally okay except for the massive dip at the end (which I have not come up with a satisfactory way of handling yet); and NOAA-16 and 17 also look okay.
Now that we’ve convinced ourselves that the offsets are real, it is time to calculate them. We obtain:
-0.136315035 -0.217185496 -0.097247497 0.215448620 -0.006319678 -0.195520508
It’s pretty obvious that these factors will reduce the trends. We get a continent-wide trend of 0.074 +/- 0.158 (compared to 0.187 +/- 0.151 from the Comiso data).
However, had we simply calculated offsets without going through the above calibration, we would have gotten:
-0.14944679 -0.16962065 -0.04562425 0.33161241 0.09104473 -0.05796546
Note that this would have even further decreased the trends – to the tune of a continent-wide average of 0.032 +/- 0.149.
Original Comiso trends (deg C/decade and 95% CI):
Peninsula 0.406552585 0.1925186
West Antarctica 0.411971312 0.2217650
Ross Ice Shelf -0.104963672 0.2206460
East Antarctica 0.225650354 0.1942384
All 0.187422507 0.1510635
Calibrated trends (deg C/decade and 95% CI):
Peninsula 0.29165083 0.2282490
West Antarctica 0.26177232 0.2502632
Ross Ice Shelf -0.12965218 0.2714344
East Antarctica 0.06001991 0.2033417
All 0.07399090 0.1572431
Here’s a plot showing my geographical groupings:
Trends about halved – very similar to what the Jeff’s got by regridding. Common theme, maybe? Suspiciouser and suspiciouser . . . but that’s enough for now. There’s a lot more in the script I posted – you can compare the main, PCA, and AWS recons as well. There’s also some single value decomposition at the end which isn’t finished yet and will be the subject of another post. Until next time, however, I will leave you with this curious plot:
The blue line is simply a slope of 1, provided for scale.
Unlabeled, one might have mistaken this for the Small Magellanic Cloud: