Shortly after, NASA published their source code on Sept 7, we started noticing puzzling discrepancies in the new data set. On Sep 12, 2007, I inquired about the changes to Hansen and Ruedy, observing that there was no notice of the apparent changes at their website:
Dear Sirs, I notice that you’ve changed the historical data for some US stations since Sep 7, 2007. In particular, I noticed that temperatures for Detroit Lakes MN in the early part of the century were reduced by nearly 0.5 deg C. These changes are subsequent to your changes in August 2007 for the changing versions. To my knowledge, there is no explanation for this most recent change and I was wondering what the reason is.
Figure 1. Difference between Sep 10, 2007 version of Detroit Lakes MN and Aug 25, 2007 version.
Thank you for your attention, Steve McIntyre
I posted on the topic on Sept 13 observing:
Since August 1, 2007, NASA has had 3 substantially different online versions of their 1221 USHCN stations (1221 in total.) The third and most recent version was slipped in without any announcement or notice in the last few days – subsequent to their code being placed online on Sept 7, 2007. (I can vouch for this as I completed a scrape of the dset=1 dataset in the early afternoon of Sept 7.)
The impact of the unreported changes was illustrated at Detroit Lakes MN using the same graphic as sent to Hansen and Ruedy. The post included the following prediction:
As you can see, Hansen has clawed back most of the gains of the 1930s relative to recent years – perhaps leading eventually to a re-discovery of 1998 as the warmest U.S. year of the 20th century.
This prediction came true quite quickly. On Sept 15, Jerry Brennan observed that the NASA U.S. temperature history had changed and that 1998 was now co-leader atop the U.S. leaderboard.
By this time, we’d figured out exactly what Hansen had done: they’d switched from using the SHAP version – which had been what they’d used for the past decade or so – to the FILNET version. The impact at Detroit Lakes was relatively large – which was why we’d noticed it, but in the network as a whole the impact of the change was to increase the trend slightly – enough obviously to make a difference between 1934 and 1998 – even though this supposedly was of no interest to anyone.
Average Impact of changing from SHAP to FILNET accounting.
Later on Sept 15, I observed:
This new leaderboard is really something else. I’m going to post on this: but if the SHAP version was what they used for the past decade, it’s a little – shall we say – “convenient” to decide in Sept 2007 that they are going to switch to the FILNET version (without announcing it on their website) and then, surprise, surprise, 1998 is now tied for the warmest year. This is going to send shivers up the spine of any readers familiar with accounting principles.
I’d been planning to write a post on this. There are undoubtedly more Climate Audit readers familiar with GAAP (Generally Accepted Accounting Principles) than at other climate websites, but it’s worth re-stating one of the fundamental GAAP principles:
Principle of the permanence of methods: This principle aims at allowing the coherence and comparison of the financial information published by the company.
Now you may say that this is “science” and accounting principles don’t apply. And my response would be that I’d expect GAAP principles to be a minimum standard for the type of climate statistics being carried out by NASA. Even if NASA climate statisticians are unaware of GAAP per se, they should be adhering to the principles. Sharp practice is sharp practice, however it is gussied up.
Hansen said that the difference between 1998 and 1934 was “statistically insignificant”. But business accountants are familiar with situations where a lot of attention is paid to numbers that may be “statistically insignificant”. I’ll give you an example. For a large corporation, the difference between a small profit and a small loss can be “statistically insignificant”, but there is a big difference in how they are perceived by the public. In some cases, unscrupulous corporations (and you can think of a few, including the most famous recent U.S. bankruptcy) will do whatever they can in terms of deferring expenses or recognizing revenue to change a reported loss into a reported profit. Accounting changes are a red flag to analysts for brokerage companies; there may be “good” reasons but the analyst needs to be right on top of the situation and they will be VERY unimpressed if a company tries to slip a change in without reporting it.
So while the difference between 1934 and 1998 may have been “statistically insignificant”. Hansen was obviously quite annoyed by the attention paid to 1934 being called the “warmest year” even in the U.S. and the change in rankings must have stuck in his craw. Was that motivation in the change from SHAP to FILNET accounting? I certainly hope not. Perhaps long before the Y2K error re-arranged things, NASA had already made long-standing plan to shift from SHAP accounting to FILNET accounting. But if this was not the case, then the timing of the change, especially with the all too “convenient” restoration of 1998 to the top of the leaderboard is certainly unfortunate.
This is precisely the type of situation that would have been avoided by NASA adhering to GAAP principles. Companies cannot change accounting procedures on a whim. Auditors will not permit companies to change methods merely to enhance reported earnings. And if a company changed accounting procedures without any disclosure, it would be viewed very seriously by regulatory agencies – whether or not the company said that it “mattered”. If the change from SHAP to FILNET accounting didn’t “matter”, then Hansen shouldn’t have done it. If it did matter, he still shouldn’t have done it right now just when he was archiving source code for the first time – and to do so without either formal disclosure or a re-statement of prior results simply boggles the imagination.
On Sept 17, Ruedy replied to my email asking that they disclose their changes, more or less refusing on the basis that the new data source could be detected in the “description of input files” in the source code.
Dear Sir,
As indicated in the description of our input files, we switched from the old year 2000 version of USHCN to the current version. The differences you noticed reflect corrections that were made by USHCN within the last six years.
Reto A. Ruedy
But this is not the same as a change statement. There’s no hint in the input file itself that they had changed the input file from what had been used previously and that the code archived on Sept 7 was NOT the code used to produce NASA results prior to Sept 7. They had not merely “simplified” the code; they had changed from SHAP to FILNET accounting. It’s also not good enough to simply slip the accounting change in with the source code. It should have been formally disclosed when the change was instituted, rather than leaving us to try to figure it out and (later disclosing it only when the change had already been discovered.)
In addition, his last sentence here – that the changes “reflect corrections that were made by USHCN within the last six years” is not correct. Both SHAP and FILNET accounts existed when Hansen et al 2001 was written. Hansen decided – for whatever reason- to use SHAP accounting. HE could have used FILNET accounting. And he decided to change in mid-September 2007. Did it “matter”? Well, it mattered enough to go to the trouble of making the change. It also – and perhaps this is sheer coincidence – mattered to the “statistically insignificant” leaderboard as 1998 is now your new co-leader/
Today NASA has attempted to cooper up this mess. At their website, they finally reported the change in accounting that we had already picked up and reported. They state:
September 2007: The year 2000 version of USHCN data was replaced by the current version (with data through 2005). In this newer version, NOAA removed or corrected a number of station records before year 2000. Since these changes included most of the records that failed our quality control checks, we no longer remove any USHCN records. The effect of station removal on analyzed global temperature is very small, as shown by graphs and maps available here.
This seems like a pretty odd description of what they appear to have done and perhaps I’ll re-visit this on another occasion. Hansen includes the following account of the Y2K error (conspicuously deleting his prior recognition of my role in identifying the error) and adding a reference to Usufruct and the Gorilla at the NASA website:
August 2007: A discontinuity in station records in the U.S. was discovered and corrected (GHCN data for 2000 and later years were inadvertently appended to USHCN data for prior years without including the adjustments at these stations that had been defined by the NOAA National Climate Data Center). This had a small impact on the U.S. average temperature, about 0.15’C, for 2000 and later years, and a negligible effect on global temperature, as is shown here.
This August 2007 change received international attention via discussions on various blogs and repetition by some other media, with no graphs provided to show the magnitude of the effect. Further discussions of the curious misinformation are provided by Dr. Hansen on his personal webpage (e.g., his post on “The Real Deal: Usufruct & the Gorilla”).
Obviously his claim that “no graphs had been provided to show the magnitude of the effect” is false. In one of my original posts on the matter, I showed graphics estimating the impact of the error on the U.S. temperature record and the distribution of errors on USHCN stations. I sent the following letter to Hansen and Ruedy today, notifying them that his statement was incorrect as follows:
Dear Sirs,
I see that you have decided to report the change in methodology as requested in my previous email. While you should have reported the change in methodology when it was made, it is better late than never.
In your new webpage, you state: ” This August 2007 change received international attention via discussions on various blogs and repetition by some other media, with no graphs provided to show the magnitude of the effect.” This is incorrect and I request that you correct this statement. On Aug 6, 2007, at Climate Audit, http://www.climateaudit.org/?p=1868 , the two graphs below were provided to estimate the magnitude of the effect. The first graph shown below estimated the impact on the U.S. temperature history at a little more than 0.15 deg C. Despite having no access to your source code, this proved to be an accurate estimate.
The next graph shown below shows the distribution of changes over the 1221 U.S. stations, which are very substantial in individual cases. Despite your professed concern for illustrating the impact of changes, you did not yourself provide any graph to show the magnitude of the changes on individual stations, nor did you even provide explicit notice on your webpage that any changes had been made.
Would you please correct the incorrect information on your webpage. This request is made pursuant to the Data Quality Act.
Yours truly,
Stephen McIntyre
A last point: as I’ve noted previously, as the classification of U.S. sites comes in, the actual GISS methodology for estimating U.S. temperatures looks a lot better than (say) the NOAA methodology. If NASA’s U.S. estimates stand up to scrutiny, that’s fine: that wouldn’t bother me a speck. I’m just trying to understand what weight can be put on which estimates. And regardless of what people may think, in a quick review of my posts, I haven’t located any posts in which I am particularly critical of NASA’s methods in the U.S., aside from the Y2K error. My position has been more: if NASA’s adjustments are right, then Parker 2006 and Jones et al 1990 etc are wrong. I had not personally criticized their lights methodology for classifying stations, preferring to see how station evaluation turned out. I have criticised poor and inaccurate disclosure, some of Hansen’s public comments and surveyed some of the data issues in the ROW (where’s Waldo?).
But I don’t think that I’ve been particularly critical of their U.S. methodology and, if the lights on-lights off criterion is a useful one for urban adjustments, that’s fine with me and I’ll be happy to acknowledge it. As noted elsewhere, that would leave many other open questions pertaining to the ROW, why there are discrepancies between NASA and NOAA, why NASA overall results are so similar to CRU results, if the individual stations are adjusted so differently etc etc.
But these matters are all quite different than (a) changing accounting systems; (b) doing so without notice; (c) archiving source code where the input file had been changed from what had been previously used; (d) making false statements on a NASA website.
UPDATE Sept 17 afternoon: Ruedy responded to my email as follows:
Thanks for bringing to our attention that the term “magnitude of effect” might be interpreted as “size” rather than “relevance”, our obvious intent. We clarified our formulation correspondingly.
They changed their website to read as follows (replacing magnitude with relevance):
This August 2007 change received international attention via discussions on various blogs and repetition by some other media, with no graphs provided to show the relevance of the effect.
Needless to say, this claim remains untrue. I sent the following letter (repeating the graphics shown above) requesting that the webapge be corrected, this time copying the Info Quality person at NASA:
Your revised webpage http://data.giss.nasa.gov/gistemp/ contains the following incorrect statement: “This August 2007 change received international attention via discussions on various blogs and repetition by some other media, with no graphs provided to show the relevance of the effect.”
This is incorrect and I request that you correct this statement. As I advised you previously, on Aug 6, 2007, at Climate Audit, http://www.climateaudit.org/?p=1868 , the two graphs below showed the relevance of the effect to U.S. temperature history and to U.S. stations.
The first graph shown below showed that the error was relevant to U.S. temperature history – a topic specifically considered in Hansen et al 2001.
The NASA website provides individual station histories, as well as U.S. and global estimates. The graph below showed the error was relevant to individual U.S. station histories.
The claim that “no graphs provided to show the relevance of the effect” remains incorrect. Once again, please correct the false statement on the NASA webpage http://data.giss.nasa.gov/gistemp/ . This request is made under the Data Quality Act.
Yours truly,
Steve McIntyre
