I’ve been foraging through ICOADS SST data for the past week and have a number of posts in progress. Here’s a diagram that I’m planning to do several comments on. It shows information on the provenance of ICOADS data between 1850, the start of the HadCRU SST record, and the present. It is very obvious that this is far from being a homogeneous network.
In recent times, the most dominant change is one that has been mentioned in recent discussion only as an aside – the change from virtually measurements being done by ship in the 1940s to the present day where measurements are done by buoys, both drifting and moored. The CRU adjustment in December 1941 coincides with the WW2 upspike in proportion of U.S. measurements. Despite the subsequent return to a more “business as usual” situation after the WW2, the Dec 1941 adjustment was not unwound. Indeed, the premise of subsequent SST estimates was that the measurement systems in 1942 remained homogeneous from then to present. The most recent CRU position is that adjustments need to be reset after WW2 to prewar levels and then re-introduced through the 1960s, after which matters remain homogeneous to the present.
Figure 1. Proportional contribution of ICOADS data 1850-1998. Left – data provided by Scott Woodruff for Woodruff et al 2005 up to 1949; right – data used in Thompson et al 2008 from 1941-1998. Red – US; pink – UK; blue – Netherlands; cyan – Germany; green – USSR; yellow= Japan;turquoise – unallocated “HSST” records; dark orange – Other (with identified nationality); orange – Other or Unknown( Including buoys).
For comparison, here is the corresponding plot for total number of measurements to 1950, which is the only information that I’ve been able to acquire so far in this format. Right is an excerpt from Rayner et al 2006, bring the same data up to the present.
Anyway, there’s lots to dwell on this diagram, which I’m going to re-visit. Notice the interesting distribution of data sources in the 19th century. Between ~1860 and WW1, there’s no American data, even though the exchange of marine data and the first conference (Brussels 1853) was inspired and organized by an American, Lt Matthew Maury, about whom more on another occasion. Surprisingly to me, the British contribution was surprisingly small, given one’s impression of the dominance of UK shipping in the 19th century. The largest assigned contributions are Dutch (blue), German(cyan). The turquoise data in the 19th century is data where early keypunching did not preserve the country of origin. Nearly 100% in the 1860s is Dutch. Who would have expected that? The first part of the 20th century is marked by huge surge in German data – reaching nearly 50% of all data at the start of WW1.
The entry of the German data – all from a single source – into the data base appears to have had an interesting story, hinted at in a laconic paragraph in Woodruff et al 2005 that caught my eye.
One of these original card decks, 192, was “punched by the German Meteorological Service [now the Deutscher Wetterdienst (DWD)] during the Nazi regime from German ship observations made during the period 1859–1939” (AWS-WB, 1958). During World War II this deck was captured by the UK, along with a land data (Kopenhagener Schluessel Synoptic Observations) deck (191) of comparable size that was captured by the US. These two decks were exchanged via a bilateral agreement reached in 1946. Approximately 37 500 original logbooks mainly for 1860–1945 are still available at DWD (Wagner, 1999).
The Dutch 19th century data has a somewhat similar vintage, described by Woodruff et al 2005:
Similarly, Dutch deck 193 (AWS-WB, 1954a), which is a predominant data source during the second half of the 19th century (see Figure 1), was the result of a keying project at the Koninklijk Nederlands Meteorologisch Instituut [Royal Netherlands Meteorological Institute] (KNMI) during 1935–1941. This deck consists primarily of Dutch (95.3%) logbooks, plus a small percentage of Swedish (3.4%) and other logbooks, most of which appear to have been lost during World War II (Wallbrink et al., 2003).
Apparently Spielberg and Lucas have wanted to do Indiana Jones 3 for a long time, but were stumped in thinking up a hook for the story – what Lucas called a MacGuffin. A MacGuffin being defined as follows:
A MacGuffin (sometimes McGuffin) is a plot device that motivates the characters or advances the story, but the details of which are of little or no importance otherwise. The element that distinguishes a MacGuffin from other types of plot devices is that it is not important what the object specifically is. Anything that serves as a motivation will do. The MacGuffin might even be ambiguous. Its importance is accepted by the story’s characters, but it does not actually have any effect on the story. It can be generic or left open to interpretation..
Surely the capture of the Hollerith punch cards stands a feat worthy of Indiana Jones and thus my suggestion to George Lucas: Indiana Jones and the Hollerith Punch Cards. I expect my royalty check any day now.
Perhaps reflecting its origins, ICOADS itself represents a remarkable intergenerational effort that seems to be governed by a different spirit than some other branches of climate science. Despite limited funding, ICOADS has placed an ENORMOUS data base online, including not just SST, but wind and numerous other details. I’ve made several inquiries to ICOADS and received cordial and informative responses. A pleasant change from Fortress CRU, where Phil Jones did not want to even disclose the identity of stations in their data set, even to the extent of resisting Freedom of Information inquiries. My impression is that there is a substantial amount of historical information that is undigitized and which ICOADS would like to get to. For whatever it’s worth, I support this.
Climate Audit 
30 Comments
Steve– I read hints of slight mods that may arise from the jet inlet-floating buoy transition in more recent times. Do you have any idea when that started? Got into full swing?
I’m working on post-war issues as well. Given the phenomenal shift from ships to buoys, it’s remarkable that so little attention has been placed on this issue. I presume that there are different types of buoys and that each one has its own quirks. The specialists indicate that buoys run a little cooler than engine inlets, but so do insulated buckets. Also there are different qualities of insulated “bucket” and substantial change in their properties since 1970, which I’ll be illustrating in a post.
I think that the various parties so far (including my earlier size up) have each only spotted one piece of the adjustments. If the reference point were engine inlets, then I think that the transition in the mid-1960s would be about halfway from uninsulated to an engine inlet reference, with another (say) 0.1-0.15 deg C adjustment in the pipeline. But rather than transitioning to an engine inlet reference point, it transitions to a buoy reference which seems to run about 0.1 deg C cooler than engine inlets. So there may well be two offsetting effects.
I’ve only seen a couple of passing references to buoy biases (e.g. Rayner et al 2006) and this is an issue that I’d like to look at.
How do they handle these huge changes in country-of-origin data? One would assume that data from a specific country would be heavily biased in where the measurements were taken. How do they link Japanese-dominated data, which one would assume would be focused on Pacific measurements, with later German-dominated data, which a presumed focus on the Atlantic?
This maybe a dumb question but if someone start to adjust data to one particular method, in this case Bucket were adjusted to inlet.
How can one justify changing the criteria of the adjustment, or shouldn’t everything be adjusted, like in this case from any method to inlet so that the data would be homogenous?
Interesting. So the US contribution presumably started by Maury in the 1850s disappeared with the Civil War, which is certainly understandable. What I find fascinating is that it didn’t resume at all until WWI, presumably because the Europeans were in the same pickle at that point. I wonder if US data was collected post-bellum, but just not used out of bureaucratic inertia. Is this data available anywhere?
It looks like a cross section of the great basin. Got discontinuities? Wow!
they need BCP in the deep blue sea, that will settle everything
I’m still puzzled by the 1960’s shift down and then up in the US ships. They all seem to go to the dark orange category of “Other (with identified nationality)” for five years or so, then back to the US column. I’m sure there’s some reasonable explanation …
While the ICOADS data has gone to buoys, I thought the HadSST2 data had gone to satellite.
How deep are the temperature sensors on the buoys?
Also, Steve, do you know if they use just night-time sea temperatures, or 24 hour sea temperatures, when figuring the averages?
I ask because the stability of ocean and atmosphere are opposite. At night, the sea is often overturning thermally, while during the day it becomes stratified, with higher temperatures near the surface. This means that during the night, once overturning is established, the depth of the sample is not too important. But during the day, sampling the surface will often give much warmer results than a deeper sample.
(In fact, one might be able to determine the depth because the closer to the surface, the higher the standard deviation … but I digress).
w.
w.
Indiana Jones 3? I want to forget “Temple of Doom” too but we are on the fourth adventure like the IPCC
I expect there is quite a bit of political and legal fog here.
Note that, right around the time of the First World War, UK and German-flagged data disappear and US records surge. This presumably reflects reflagging of vessels to claim neutrality and avoid being seized or sunk. [Many of the UK records from ~1913-1917 could be at the bottom of the ocean, I suppose.] These could easily be data taken on the same ships, by the same crew members, but under different claimed nationality.
The 1945-1950 drop in US records also looks like a reflagging of US vessels to UK post-war.
British shipping takes off around the time the Suez canal was opened [1869]. There appears to be a loss in UK shipping around the time of the Suez crisis [1956]. UK shipping also dropped significantly around the time of the Boer War [1898], which was very unpopular; were there boycotts ?
US records also take off from zero right about the time that the 16th Amendment was ratified [1916], authorizing the collection of income tax. It makes me wonder if, before the income tax, the US had some onerous tax on ship tonnage that incentivized reflagging. The Panama Canal also opened at this time [1914].
The same could be said for the 1964-1968 period, where it appears ~95% of US vessels have been reflagged dark orange [Liberia ?]. Was some screwy law in effect during that period; laws often cause shipowners to reflag.
Note also that USSR records pre-date the founding of that state – the earlier records are presumably russian imperial records. Soviet shipping, or at least the reporting of it, takes off about the time Stalin dies and Khrushchev takes over.
Why did the proportion of Japanese shipping drop off before 1920 ? Japan had no enemies in the Pacific at this time. Did Japanese shipping drop off or, with the opening of the Panama canal and burst of western capitalism, did western shipping increase so much as to make Japanese shipping appear proportionately smaller ? Or did western mariners start measuring with greater frequency after the Titanic sank ?
A second graph with same colors showing absolute number of records might be interesting.
I replaced my graphic which contained a coloring error. In the 19th century, there is a large data set (HSST) for which early generation keypunching did not record country of provenance. It looks like this data comes from the major sources. I’ve also added a plot of aggregate measurements to 1950, which is the data that I’ve been able to obtain so far.
This question is highly unscientific, but it it possible/easy to forget country and apparatus and location for a moment and derive a mean temp for each decade, all measurements, just as a curtain raiser? If it’s a flat liner, it might point to less effort needed. (But Life being Life, it won’t be, will it?)
Woodruff et al 2005:
The American merchant marine virtually disappeared after 1850, and Britain had half the world’s merchant tonnage in the last half of the 19th c. and up to 1914.
The numbers of ships sunk in World War I is astonishing. I don’t have the numbers handy, but it was many thousands. These were largely, of course, British.
If there is any justification for making an adjustment based on US activity in World War II, then 1942 is way too early to make it.
Steve-Maybe because I’m a history buff, i’m really enjoying following this discussion and think your unravelling fascinating stuff here!
Isn’t the question not so much the national registration of the shipping- I believe up to 90% up pre-WW1 international shipping was UK registered – but the shipping lanes that those vessels traversed? My understanding is that though UK shipping was very international in in the seas it crossed, other nationally registered shipping, German, Japanese and US, crossed only limited trade routes. (UK free-trade policy gave it an advantage in the shipping market). The high percentage of non-UK shipping that you have shown in the data would indicate that the temperatures for only a restricted area of the oceans was actually tested. As most shipping routes never went near large areas of ocean- South Atlantic, South Pacific etc. – wouldn’t this mean COADS historic data had a very limited global coverage?
Yes, you are very much correct. However, a scientific hoax by definition does not require stringent data sampling methods which may spoil and dispel the illusion that grossly irregular and inadequate data sampling is somehow supposed to be reliable enough to represent an historical measurement record. If the IPCC sees no problem with losing major cities and their inconvenient temperature records in South America and other locations around the world, how much more unrealistic do you suppose IPCC supporters expect it to be for anyone to actually require adequate samples from wide regions of the Earth’s unpopulated oceanic thermal masses [a rhetorical question of course]? If the IPCC and its supporters expect to be taken seriously, they must demonstrate with some truly solid scientific evidence that such sampling methods adequately represent the thermal condition and capacity of the hydrosphere, elsewise all conclusions without such a foundation amount to nothing more than scientific nonsense and scientific fraud when applied to unlicensed planetary engineering.
A brief introduction to the changing activities of the U.S. merchant fleets and U.S. Merchant Marine describes how economics and changing governmental policies resulted in wide changes in the sizes and activities of these merchant fleets. See:
Kesteloot, Robert W. The U.S. Flag-Merchant Marine: A Century in Review. Navy League of the United States. http://www.navyleague.org/seapower/us_flag_merchant_marine_a_century_in_review.htm
Astute readers may note and react to the obvious parallel between the boom and bust governement efforts to regulate a successful merchant fleet and a successful planetary climate.
An inconsequential point – the words ‘..what Lucas called a MacGuffin..’ suggest to me that this is some special Lucas usage.
In fact, the word ‘MacGuffin’ to denote a plot device was first coined by Hitchcock and a writer friend of his in the 1930s, and by now must be standard usage amongst all film buffs who consider movie plots.
SteveMc and StuartR:
I just can’t resist. There seems to be some confusion as to the timeline of Indiana Jones movies. There are now 4, in the following order: Raiders of the Lost Ark, The Temple of Doom, The Last Crusade, and The Kingdom of the Crystal Skull.
With all that being said, I agree that this data would make a noble quest for the anthropologist/adventurer. Especially if it can be found without adjustments!
Re # 16 D Patterson
No comment about a couple of recent no-nos by USAF?
I haven’t followed these very intersting discussions on SST in enough detail to know if this has already been done. But what does the SST and total global (SST plus land) look like without makeing any adjustments?
The reason I ask is that when you look at the multi-coloured graph at the top of this entry, I would say there is too much uncertainty to make any assumptions about when or how to introduce ‘corrections’.
Regarding?
Re# 16 D.Patterson
Thanks for the comments and perhaps I was being a bit naive in expecting higher standard than what we appear to be getting. On reflection, the inconsistencies of the ocean records are simply a reflection of what has happened with the land records – bristlecone proxies et al- except it’s writ large.
With regards to the change over to flags of conveniece ( Re# 10 Edwards ) and Government attempts to manage their merchant marine, I believe only a few Governments (i.e Japanese )succeeded in this endevour. The Greeks were very successful in attracting this business post WW11. When a ship was “de-flagged” the officers still tended to be of the original nationality: I suspect there was little change in data quality post de-flagging.
It would be interesting to see what your impression is like after you’ve spoken to some of the folks who had routine cause to board, inspect, or survey these vessels and their crews. Among the many reasons why ships were reflagged is also the one in which ship owners regarded the U.S. maritime regulations and crew safety requirements to be too onerous.
D. Patterson (23)
I agree, re-flagging was probably done for convenience of the owners, either for taxes or avoidance of certain laws.
But remember, just as the surface stations were set to monitor local contitions and not perform climate research, then so were the SST temps (taken to determine settings for ship performance, or to monitor weather, and not perform climate research).
A change in flag would not have changed the reasons for the measurements of SST. They may have changed the method used (ship crews would have performed the measurements they were most familiar with), but not the reason for taking them.
The parts of the ocean that were sampled changed considerably in 1914, when the Panama Canal opened. And in 1867, when the Suez Canal opened, but I guess the number of samples was not large in the ’60s.
Re # 21 D Patterson
As I read, the Chief of Staff, Mosely, and the Secretary for Defense, Wynne, were asked to resign the other day. Some of the reasons?
A change in flag and/or a change in ownership often did change the reasons for measuring sea surface temperatures (SST), due to the differences in attitudes towards participation in the Voluntary Observing Ships (VOS) projects Likewise, changes in command had its effect upon crew performance. Since only a fraction of the world’s fleet of merchant ships ever participated in the VOS projects contributing to the ICOADS database, merchant ships participating in the projects constantly changed, and command and crews constantly changed, the methods used and accuracies used to collect the SST data correspondingly changed in accuracy and reliability in unobserved and unreported ranges of error.
Ships’ crews are in a state of constant change and retraining, and their performance standards with respect to the accuracy of SST measurements have only been hypothesized and estimated. There is no present means of obtaining an actual measure of performance accuracy applicable to the majority of the ICOAD observations of SST.
The substantive reason for the resignations was Gates’ desire to end Air Force opposition to an eroding investment in the F-22 Raptor program and other programs devoted to strategic deterrence versus short-term tactical operations with UAV aircraft in the GWOT. The performance related incidents supplied good reasons to request the resignationss without getting into the real and greater substantive issues. Although this is all related to the problems typically found in budget battles and performance issues related to limitations of resources, it appears to be veering sharply off-topic from the issue of competence of an international political organization to plan and conduct planetary engineering projects and determining sea surface temperatures as a measure of the planetary heat condition and capacity.
Re # 28 D. Patterson
Yes, I agree with you that this is OT – to an extent. You raised the “boom and bust” point of government management.
As one who worked mainly in private enterprise, I have opinions and examples about how Government regulation of private enterprise is/is not a good thing and how much we should trust highly placed official bodies, be they IPCC or whomever. A priori, I see no reason to discriminate on quality grounds between SST temps from navies and private operators, if there developed an inclination to do so.
The persistence of bodies like ICOAD deserves admiration. I equate them to those who have saved ancient and antique treasures through periods of conflict. It will be valuable to see how much SST can be reconstructed with confidence.
Climate Audit is becoming increasingly important among those I know. The frequecy of prima facie errors in matters raised by CA is greater than I am accustomed to.
ICOAD certainly has a lot to offer in the way of historical weather records, which is why I have often urged its use in this forum for many years past. However, I am more than a little concerned by any inappropriate usage of these records to sustain arguments while blithely disregarding some very serious limitations in their accuracy and reliability. The observational weather records contain many entirely bogus observations in addition to the observations which have systematic errors. Present users are failing to heed warnings about such problems with these bogus records and their potential effects on the analyses and adjustments of the data series. Simply put, people are formulating conclusions while using data that was faked at the date of collection in addition to any additional faults added at the present time. This is why I must continue to treat these efforts to wring fractional differences out of such faulty data with a certain degree of disgust at such foolishness. My reaction is not unlike the comments made by some coal miners I know who have voiced their contempt of some young mining engineers who fail to understand how a mining tunnel doesn’t end-up in quite the place the mining engineer wrongly believed it should. Real world operations often do not conform to the rules and expectations of people who like to believe they are experts, while the people doing the day to day work know better from experience.
One Trackback
[…] However, this previously non-existent adjustment is important to HadSST3. There is evidence (discussed at CA here) of buckets being widely used into the 1970s, despite the Pearl Harbour assumption. By distinguishing between uninsulated buckets and insulated buckets and providing for a changeover from uninsulated buckets (pre-WW2) to insulated buckets by the 1970s, most of the effect of a gradual changeover is allocated prior to 1975, thus limiting changes after the 1970s, where there is also a satellite record that would need to be reconciled. While there is still an effect for changeover from insulated buckets to engine inlets after the 1970s, there is also evidence that the introduction of buoys in this period results in an offsetting cold bias. I referred to these issues in the final post of my 2008 series on Thompson et al 2008 here. I did one more post on SST at the time: a short consideration of the ICOADS data set here. […]