In the Guardian debate, George Monbiot’s opening question (made in good faith on his part) pertained to CRUTEM, George noting that the inquiry had been able to derive a CRUTEM-like result from GHCN data and challenging me that this had somehow rebutted my “crusade” on this point.
I tried to deal with this as quickly as I could, since I did not want to waste an already short 5 minutes to deal with disinformation. My answer – which surprised Monbiot – was that CRUTEM had been little more than a passing interest at Climate Audit on which I’d seldom commented. And that Muir Russell’s finding on the triviality of CRU’s temperature unit simply endorsed a point previously made at Climate Audit. This answer seemed to baffle George and others.
Unfortunately, Monbiot and others had uncritically accepted disinformation from the Muir Russell inquiry, which, on this point (as on some others), instead of examining (with citations) actual criticisms from sources like Climate Audit, preferred instead to construct its own allegations which, in this case, they described as “broad allegations which are prevalent in the public domain”. Lucia has often criticized such Gavinesque behavior in other contexts.
My long-standing position on CRUTEM was that CRU’s obstruction of data requests was most likely due to its desire to conceal that it did so little work on quality control; that the CRU result could be derived so trivially that, in effect, CRU no longer served any useful function in this field. Long before Climategate, I’d recommended that CRU’s responsibilities in this field be transferred to the UK Met Office and that the US Department of Energy re-allocate its funding in this area to improvements at GHCN – a point that should be considered carefully in the US DOE review of their funding of CRU (reported by Jonathan Leake here.)
At the Guardian panel, I observed that CRUTEM was an almost microscopically small issue in the Climategate emails – Climategate was about the Hockey Stick and its handling by IPCC, not CRUTEM. CRUTEM was mentioned in only 25 emails and, even then, often passim.
I’ll review some past CA posts to provide support for this.
In early 2007 here, I’d observed that the HadCRU series for gridcell 57N 77E (containing the single Siberian station, Barabinsk) could be derived from a simple anomaly calculation from a GHCN version of Barabinsk. At the time, we didn’t know what stations CRU used, but, in this case, I observed that the CRU calculation was straightforward – unlike GISS, which had all sorts of weird smoothing and adjusting, which were then a topic of interest. This post contains some interesting plots of differences between various various versions for the gridcell and station – the understanding of these differences has underpinned the desire to examine data as used by the various agencies.
In late 2008, long before my own FOI requests for CRU station data, I discussed CRU calculations in more detail here, concluding with the observation that “if, like GISS, they are doing nothing other than trivial sums on GHCN data, one feels that the money would be better spent on beefing up QC and data collection at GHCN.”
The reverse engineering of CRUTEM3 looks almost pathetically easy given that we’ve already waded through step 0 of GISS, where they collate different GHCN versions (dset0) into a single station history (dset1.) CRU doesn’t have the bewildering sequence of smoothing operations that Hansen uses at multiple stages (though Hansen, mercifully, doesn’t use Mannian butterworth smoothing).
To my knowledge, unlike GISS, CRU does not make the slightest attempt to adjust for UHI, relying instead of articles like Jones et al 1990 purporting to show that UHI doesn’t “matter”.
We can already emulate GISS step0 – not that it makes any sense, but it provides a benchmark. Here’s all that seems to be necessary to produce a gridded CRUTEM3 series given a dset1 data set. First, create an anomaly-version of the series. I have a simple function anom on hand and this could be done as follows:
Then one could make an average of dset1 series within gridcell i as follows, where info is an information dataset in my usual style containing for each station, inter alia, its lat, long and gridcell number (called “cell” here):
for (i in 1:2592) grid[,i]=apply(dset1.anom[,info$cell==i],1,mean,na.rm=T)
This would yield the CRUTEM3 series. My guess as to why they don’t
want to show their work is because they probably use hundreds of line of bad Fortran code to do something that you can do in a couple of lines in a modern language. Anyway, I’ll experiment with this at some point, but this is my hypothesis on all that’s required to emulate CRUTEM3. CRU has been funded by the US DOE; if, like GISS, they are doing nothing other than trivial sums on GHCN data, one feels that the money would be better spent on beefing up QC and data collection at GHCN.
We re-visited this issue in the Mole post last summer. I observed in a comment (along the lines of my 2008 post):
Nowhere have I encouraged readers to expect any smoking guns in this data set. Quite the opposite. My own best guess as to why they are so obstructive about the data is the specific commercial interest of CRU. My guess is that they spend negligible time on quality control, but derive a lot of funding for a prestigious data set and use the funds for other purposes. They don’t want anyone to see how simplistic their analysis is and how negligible their quality control. Nothing more, nothing less. (But that’s just a guess. The real reason may be different again.)
Reader Adam observed:
The whole CRUTEM / HadCRUT gridded series can be easily reproduced for the most of the globe with the GHCN dataset. I’ve tried it for some gridcells and it worked.
This is not actually true, though it holds for some gridcells (as I’d observed for the Barabinsk gridcell.) For example, HadCRU includes SST data, which is not in the GHCN land data set. In addition, Jones has his own 1961-1990 “normals” that he uses for standardization and an exact replication of CRUTEM cannot be accomplished without these “normals”, though the calculation can be approximated using freshly calculated normals. (Oddly enough, I have a copy of the CRUTEM2 normals from my 2002 correspondence with Jones – before I’d been blacklisted because of the MM2003 criticism of MBH98). In response to Adam, I observed:
I agree with your comments. Like you, I believe that 95% of CRU is obtained from GHCN, with a very few non-GHCN sources, of which Austria is one (Norway, Sweden, Denmark are others.) Like you, I believe that they do relatively trivial manipulations of GHCN data. AS I’ve said elsewhere, that is my best guess as to the secret that they don’t want exposed and the only commercial interest that they are protecting.
Also see my post from earlier this year discussing the “end of CRUTEM” and the desireability of this responsibility being taken over by the Met Office, a post in which I review earlier comments to this effect going back a few years.
Jones has spent much of his academic career as a sort of temperature accountant. Commencing in the early 1980s, he collected station data and compiled averages – a useful enterprise, but surely no more than accounting.
Muir Russell’s second of three “broad allegations which are prevalent in the public domain”:
That CRU adjusted the data without scientific justification or adequate explanation. Some allegations imply that this was done to fabricate evidence for recent warming.
I don’t know who they had in mind here, as they’ve followed the Gavin Schmidt practice of not providing a citation. And perhaps somebody somewhere has made an allegation in this form. But this is not an allegation that was made at Climate Audit. My own surmise was not that CRU had adjusted the data, but that they hadn’t adjusted the data for UHI – a surmise that has been verified.
The criticism from Climate Audit was that (1) CRU provided their station data as collated to “friends” but not to potential critics; and (2) that their excuses for not providing station data were what one London reporter (not Jonathan Leake) described to me as “deliberately deceptive”.
Muir Russell did not directly address either issue. Instead, they re-framed both questions.
Postscript: In their Appendix 7, Muir Russell say that they were able to make a concordance of 90% of CRU stations to GHCN stations despite the lack of such a concordance by CRU. This is a lower percentage than the concordance (95.6%) that I’d placed online in December 2007.
Muir Russell stated:
31. The Review Team was able to match 90% of stations given in the CRU list to GHCN (see Appendix 7). CRU has stated in a written submission  that that the remaining 10% can be obtained from other sources including the NMOs. Thus substantial work is required to take the CRU published list and assemble 100% of the primary station data from global repositories and NMOs. We make a recommendation for the future below.
When Willis Eschenbach managed to get the list of CRU stations as used, I did semi-automated matching of CRU information to GHCN information, discussing the matter here and posted up my concordance in Dec 2007 here, in which I’d matched 95.6% of CRU stations to GHCN sources.