A new HadSST3 version has been recently published. It starts the process of unwinding Folland’s erroneous Pearl Harbour bucket adjustment, an adjustment that has been embedded in HadSST for nearly 20 years.
Folland’s erroneous adjustment had been originally criticized at CA in 2005 here and further discussed at length in March 2007 at CA here, a post in which I observed that no climate scientist had made any attempt to validate Folland’s bizarre adjustment and that correcting Folland’s error (through a more gradual and later changeover to engine inlets than the worldwide overnight change that Folland had postulated after Pearl Harbour) would have the effect of increasing SST in the 1950s, in turn, potentially eliminating or substantially mitigating the downturn in the 1950s that was problematic for modelers.
However, not until Thompson et al 2008 (submitted Jan 2008; published May 2008) was the problem with the Folland adjustment clearly acknowledged by the “community”. The importance of Thompson et al in resolving the problems arising from the Folland adjustment were credited by Susan Solomon and Phil Jones in the commentary accompanying the Nature article.) Both lead author David Thompson and co-author Mike Wallace, though very prominent climate scientists, had negligible (or no) publishing history on the topic; as one commenter at James Annan’s blog put it, they came out of “left field”. Thompson was an ozone specialist. The other co-authors, John Kennedy of the Hadley Center and Phil Jones of CRU, were, of course, actively involved in the field.
Now over three years later, in a new SST edition (HadSST3), the Hadley Center has accepted and implemented Thompson et al’s criticism of Folland’s Pearl Harbour adjustment. Instead of implementing an overnight changeover to engine inlets in December 1941 as before, the changeover is now phased in through the mid-1970s. This results in changes to SSTs between 1941 and ~1975.
In the new edition, they introduce a new adjustment for an intermediate changeover from uninsulated buckets to insulated buckets. Under the Pearl Harbour assumption, insulated buckets had either been disregarded (because of the assumption that a changeover to engine inlets had been complete at Pearl Harbour) or were held to be of only marginal significance and thus not included in the calculations (the apparent position of Rayner et al 2006).
However, this previously non-existent adjustment is important to HadSST3. There is evidence (discussed at CA here) of buckets being widely used into the 1970s, despite the Pearl Harbour assumption. By distinguishing between uninsulated buckets and insulated buckets and providing for a changeover from uninsulated buckets (pre-WW2) to insulated buckets by the 1970s, most of the effect of a gradual changeover is allocated prior to 1975, thus limiting changes after the 1970s, where there is also a satellite record that would need to be reconciled. While there is still an effect for changeover from insulated buckets to engine inlets after the 1970s, there is also evidence that the introduction of buoys in this period results in an offsetting cold bias. I referred to these issues in the final post of my 2008 series on Thompson et al 2008 here. I did one more post on SST at the time: a short consideration of the ICOADS data set here.
The new HadSST3 dataset still contains some seemingly arbitrary assumptions. They assert that 30% of the ships shown in existing metadata as measuring SST by buckets actually used engine inlet and proceed to reallocate the measurements on this assumption:
It is likely that many ships that are listed as using buckets actually used the ERI method (see end Section 3.2). To correct the uncertainty arising from this, 30+-10% of bucket observations were reassigned as ERI observations. For example a grid box with 100% bucket observations was reassigned to have, say, 70% bucket and 30% ERI.
The supposedly supporting argument at the end of Section 3.2 is as follows:
It is probable that some observations recorded as being from buckets were made by the ERI method. The Norwegian contribution to WMO Tech note 2 (Amot ) states that the ERI method was preferred owing to the dangers involved in deploying a bucket. This is consistent with the rst issue of WMO Pub 47 (1955), in which 80% of Norwegian ships were using ERI measurements. US Weather Bureau instructions (Bureau ) state that the \condenserintake method is the simpler and shorter means of obtaining the water temperature” and that some observers took ERI measurements \if the severity of the weather [was] such as to exclude the possibility of making a bucket observation”. The only quantitative reference to the practice is in the 1956 UK Handbook of Meteorological Instruments HMSO  which states that ships that travel faster than 15 knots should use the ERI method in preference to the bucket method for safety reasons. Approximately 30% of ships travelled at this speed between 1940 and 1970.
This adjustment would reduce the difference between HadSST2 and HadSST3, though the size of the impact was not reported in Kennedy et al 2011. I think that it is reasonable to hope for more conclusive documentary support for overwriting actual data particularly given that the changes described in Kennedy et al 2011 arise from unwinding previous adjustments made without documentary support,
Another somewhat quirky methodology of Kennedy et al 2011 is reported as follows:
Some observations could not be associated with a measurement method. These were randomly assigned to be either bucket or ERI measurements. The relative fractions were derived from a randomly-generated AR(1) time series as above but with range 0 to 1 and applied globally.
I have no idea at present why one would do things this way or what its effect is. It seems like an odd methodology.
Overlooked thus far is the impact of the HadSST revision on the relationship between HadSST and CRUTEM, both said to be “independent” series. The three series – HadSST2, HadSST3 and CRUTEM, are shown in the figure below from 1940 (the point of departure) to 2006:
Figure 1. Global HadSST2(black), HadSST3 (red) and CRUTEM (green).
Although the results of these series are often said to be mutually supporting, between 1975 and 2006, CRUTEM has increased substantially more than HadSST, with both HadSST versions very similar in this period: CRUTEM: 0.243 deg/decade; HadSST: 0.135 deg/decade. Over the 1940-2006 period, the difference in trends is 0.082 deg?decade, resulting in a cumulative difference of 0.41 deg C (56 years at 0.082 deg/decade).
Over the 1940-2006 period illustrated in the above figure, the HadSST trend is reduced by 35% from 0.074 deg/decade (HadSST2) to 0.048 deg/decade (HadSST3). The decrease in 1950-2006 trend from HadSST2 to HadSST3 is 29.5%, a value that is “remarkably similar” to the figure of 30% postulated in 2008 by Pielke Jr. (Both Pielke’s calculation and my related calculations were restricted to the effect of unwinding the Pearl Harbour assumption and were thus not apples-to-apples to the present HadSST2-HadSST3 differential, which incorporates other adjustments.) Given the closeness of Pielke’s 30% to the actual decrease in 1950-2006 trend from HadSST2 to HadSST3, it is remarkable that Schmidt singled this estimate out for particular contumely (while not criticizing the 20-year failure of IPCC scientists to unwind the erroneous Folland method.)
Rather than reporting the change in trend for the HadSST series that had been illustrated in the first two figures of the realclimate post (SST series had been at issue in Thompson et al 2008 and the subsequent discussion), Schmidt estimated changes in trend for HadCRU by combining the HadSST changes with unchanging CRUTEM (70% HadSST; 30% CRUTEM), only reporting the decrease in land-and-ocean trend (and not the decrease in SST trend.) On a 70% basis, the 29.5% decrease in HadSST trend equates to about 20% on HadCRU basis (Schmidt reported 17%.)
Which is the “right” way of presenting the calculation: SST or combined land-and-ocean? There are pros and cons to each calculation. Given that CRUTEM and HadSST are held to be independent series, I think that it is definitely important to visualize the impact of the change from HadSST2 to HadSST3 against CRUTEM, particularly since the discrepancy between the two series increases with the HadSST3 revisions. The original blog discussion at CA was entirely focused on SST (CRUTEM never being mentioned), though one graphic used HadCRU though the discussion was about HadSST. Be that as it may, in a discussion of HadSST, even at realclimate, it wouldn’t have been out of place for Schmidt to actually mention the decrease in 1950-2006 trend from HadSST2 to HadSST3: 29.5%.