I’ve been foraging through ICOADS SST data for the past week and have a number of posts in progress. Here’s a diagram that I’m planning to do several comments on. It shows information on the provenance of ICOADS data between 1850, the start of the HadCRU SST record, and the present. It is very obvious that this is far from being a homogeneous network.
In recent times, the most dominant change is one that has been mentioned in recent discussion only as an aside – the change from virtually measurements being done by ship in the 1940s to the present day where measurements are done by buoys, both drifting and moored. The CRU adjustment in December 1941 coincides with the WW2 upspike in proportion of U.S. measurements. Despite the subsequent return to a more “business as usual” situation after the WW2, the Dec 1941 adjustment was not unwound. Indeed, the premise of subsequent SST estimates was that the measurement systems in 1942 remained homogeneous from then to present. The most recent CRU position is that adjustments need to be reset after WW2 to prewar levels and then re-introduced through the 1960s, after which matters remain homogeneous to the present.
Figure 1. Proportional contribution of ICOADS data 1850-1998. Left – data provided by Scott Woodruff for Woodruff et al 2005 up to 1949; right – data used in Thompson et al 2008 from 1941-1998. Red – US; pink – UK; blue – Netherlands; cyan – Germany; green – USSR; yellow= Japan;turquoise – unallocated “HSST” records; dark orange – Other (with identified nationality); orange – Other or Unknown( Including buoys).
For comparison, here is the corresponding plot for total number of measurements to 1950, which is the only information that I’ve been able to acquire so far in this format. Right is an excerpt from Rayner et al 2006, bring the same data up to the present.
Anyway, there’s lots to dwell on this diagram, which I’m going to re-visit. Notice the interesting distribution of data sources in the 19th century. Between ~1860 and WW1, there’s no American data, even though the exchange of marine data and the first conference (Brussels 1853) was inspired and organized by an American, Lt Matthew Maury, about whom more on another occasion. Surprisingly to me, the British contribution was surprisingly small, given one’s impression of the dominance of UK shipping in the 19th century. The largest assigned contributions are Dutch (blue), German(cyan). The turquoise data in the 19th century is data where early keypunching did not preserve the country of origin. Nearly 100% in the 1860s is Dutch. Who would have expected that? The first part of the 20th century is marked by huge surge in German data – reaching nearly 50% of all data at the start of WW1.
The entry of the German data – all from a single source – into the data base appears to have had an interesting story, hinted at in a laconic paragraph in Woodruff et al 2005 that caught my eye.
One of these original card decks, 192, was “punched by the German Meteorological Service [now the Deutscher Wetterdienst (DWD)] during the Nazi regime from German ship observations made during the period 1859–1939” (AWS-WB, 1958). During World War II this deck was captured by the UK, along with a land data (Kopenhagener Schluessel Synoptic Observations) deck (191) of comparable size that was captured by the US. These two decks were exchanged via a bilateral agreement reached in 1946. Approximately 37 500 original logbooks mainly for 1860–1945 are still available at DWD (Wagner, 1999).
The Dutch 19th century data has a somewhat similar vintage, described by Woodruff et al 2005:
Similarly, Dutch deck 193 (AWS-WB, 1954a), which is a predominant data source during the second half of the 19th century (see Figure 1), was the result of a keying project at the Koninklijk Nederlands Meteorologisch Instituut [Royal Netherlands Meteorological Institute] (KNMI) during 1935–1941. This deck consists primarily of Dutch (95.3%) logbooks, plus a small percentage of Swedish (3.4%) and other logbooks, most of which appear to have been lost during World War II (Wallbrink et al., 2003).
Apparently Spielberg and Lucas have wanted to do Indiana Jones 3 for a long time, but were stumped in thinking up a hook for the story – what Lucas called a MacGuffin. A MacGuffin being defined as follows:
A MacGuffin (sometimes McGuffin) is a plot device that motivates the characters or advances the story, but the details of which are of little or no importance otherwise. The element that distinguishes a MacGuffin from other types of plot devices is that it is not important what the object specifically is. Anything that serves as a motivation will do. The MacGuffin might even be ambiguous. Its importance is accepted by the story’s characters, but it does not actually have any effect on the story. It can be generic or left open to interpretation..
Surely the capture of the Hollerith punch cards stands a feat worthy of Indiana Jones and thus my suggestion to George Lucas: Indiana Jones and the Hollerith Punch Cards. I expect my royalty check any day now.
Perhaps reflecting its origins, ICOADS itself represents a remarkable intergenerational effort that seems to be governed by a different spirit than some other branches of climate science. Despite limited funding, ICOADS has placed an ENORMOUS data base online, including not just SST, but wind and numerous other details. I’ve made several inquiries to ICOADS and received cordial and informative responses. A pleasant change from Fortress CRU, where Phil Jones did not want to even disclose the identity of stations in their data set, even to the extent of resisting Freedom of Information inquiries. My impression is that there is a substantial amount of historical information that is undigitized and which ICOADS would like to get to. For whatever it’s worth, I support this.