Climate Audit

Steve McIntyre

You’ve demonstrated nada. The RVose file is form last February and is irrelevant.

I corresponded with Vose today and in that email he didn’t know what the problem was.

Please stop congratulating yourself prematurely, DC. If you show a data set with valid October data fine, but so far you’ve thrown spitballs. Please check out the data.

Deep Climate

Yes, I see that it has no data for 2008, so I withdraw that last one.

But I think you still need to consider that a file identified as FD might actually be FD, and one described as anomaly would contain anomalies. Right? Why not try it for the gridcell in question?

• James Lane
  Posted Nov 21, 2008 at 1:44 AM | Permalink

  Re: Deep Climate (#4),

  But I think you still need to consider that a file identified as FD might actually be FD, and one described as anomaly would contain anomalies. Right? Why not try it for the gridcell in question?

  Hey DC. Here’s an idea, why don’t you try it and report back?

• KimberleyCornish
  Posted Nov 21, 2008 at 5:24 AM | Permalink

  Re: Deep Climate (#4),

  Would you mind being a little more specific about precisely what you are withdrawing? It seems to me that you accused Steve of using the wrong data file, then you referenced what you mistakenly thought was the correct file that Steve SHOULD have used, only to find that what you referenced was dodgy. Is this a correct account of it? One would think that if you can produce the file you think Steve should have used, then discussion can proceed. If you can’t, then your credibility is somewhat suspect. The file removal shenanigans are not something either innocent or separate from your participation in this blog discussion. That is to say, pleaase either put up or shut up. That way bandwidth is not wasted, but made available for those who want to engage in serious discussion.

• RomanM
  Posted Nov 21, 2008 at 11:27 AM | Permalink

  Re: Deep Climate (#4)

  But I think you still need to consider that a file identified as FD might actually be FD, and one described as anomaly would contain anomalies. Right? Why not try it for the gridcell in question?

  Why is it that the most arrogant posters are also usually the least likely to check that their statements are at least based on correct facts? First, read the readme file carefully (bold mine):

  These data sets contain gridded temperature anomalies for three parameters (mean,minimum, and maximum temperatures) from the GHCN V2 monthly temperature data sets.
  …

  There are two options: grid_1880_YYYY.dat.gz, where YYYY is the current year and the values are calculated using the first difference method, and anom-grid-1880-current.dat.gz which uses the “anomaly” method.

  …

  We calculated these anomalies with respect to the period 1961 – 1990 using the First Difference Method, an approach developed to maximize the use of available station records (see, e.g., Peterson et al., 1998, ‘The First Difference Method: Maximizing Station Density for the Calculation of Long-term Global Temperature Change’, Journal of Geophysical Research). The
  First Difference Method involves calculating a series of calender-month differences in temperature between successive years of station data (FDyr = Tyr – Tyr-1). For example, when creating a station’s first difference series for mean February temperature, we subtract the station’s February 1880 temperature from the station’s Februrary 1881 temperature to create a February 1881 first difference value. First difference values for subsequent years are calculated in the same fashion by subtracting the station’s preceding year temperature for all available years of station data.

  For each year and month we sum the ‘first difference’ value of all stations located within the appropriate 5 X 5 degree box and divide by the total number of stations in the box to get an unweighted first difference value for each grid box. We then calculate a cumulative sum of these gridded first difference values for all years from 1880 to 1998 to produce a time series for each grid box. The cumulative sum is calculated for each grid box and each month of gridded first difference data independently through time. Each grid box time series is then adjusted to create anomalies with respect to the period 1961 – 1990.

  Still not good enough for you? OK, let’s check for the “gridcell in question”. We will take the data as downloaded by Steve Mc in the thread Anomalous in Finland. Fortunately, I managed to download the NOAA data set before the magic disappearing act started.
  Do some R work:

  swede = ts.union(station1,station2,noaa[,329])
  #extract October
  #Note NOAA values for October are all NAs before 1992
  swede.oct = window(swede,c(1880,10), frequency=1)

  #form first difference sequence for all three series
  #(including the NOAA series)
  swede.diff = diff(swede.oct)

  #average the two station series
  swede.dave=(swede.diff[,1]+swede.diff[,2])/2

  #display the results from 1991 to 2008
  window(ts.union(swede.diff,swede.dave), 1991)

  Time Series:
  Start = 1991.75
  End = 2008.75
  Frequency = 1
  swede.diff.station1 swede.diff.station2 swede.diff.noaa[, 329] swede.dave
  1991.75 -0.6 0.0 NA -0.30
  1992.75 -8.1 -8.2 NA -8.15
  1993.75 4.3 4.7 4.50 4.50
  1994.75 2.5 1.9 2.20 2.20
  1995.75 -0.5 2.3 0.90 0.90
  1996.75 3.0 0.5 1.75 1.75
  1997.75 -1.8 -4.0 -2.90 -2.90
  1998.75 0.3 2.3 1.30 1.30
  1999.75 1.3 0.8 1.05 1.05
  2000.75 2.4 2.6 2.50 2.50
  2001.75 -3.8 -3.1 -3.45 -3.45
  2002.75 -3.1 -4.5 -3.80 -3.80
  2003.75 2.6 3.4 3.00 3.00
  2004.75 0.5 0.1 0.30 0.30
  2005.75 1.6 2.5 2.05 2.05
  2006.75 -3.1 -4.2 -3.65 -3.65
  2007.75 3.9 4.7 4.30 4.30
  2008.75 -3.1 -1.8 -2.45 -2.45

  You will note that the mean of the two “first differences” of the temperature series matches the first diffs of the original NOAA series. This indicates the latter had to be anomalies and not what was incorrectly claimed by DC.

  So why are the results so far out of line from the observed data? The appears to be created by those MIN (missing in noaa) values prior to 1992 combined with the fact that the base year from which the first of the differences was calculated by NOAA was the coldest October on record for both stations. It is not clear how NOAA deals with missing values in constructing the anomalies from the first diffs, so I won’t try now. My strong suspicion is that Steve’s 6.66 degree difference will be explained by this fact.

  • fFreddy
    Posted Nov 21, 2008 at 12:20 PM | Permalink

    Re: RomanM (#22),

    Why is it that the most arrogant posters are also usually the least likely to check that their statements are at least based on correct facts?

    Arrogant – from the latin, “a rogare” – “not asking”

Mike Bryant

It’s getting pretty deep around here. Time to get a shovel.

Sinan Unur

For everyone’s reference, you can see the history of changes to the contents of ftp://ftp.ncdc.noaa.gov/pub/data/ghcn/v2/grid/ at http://www.flickr.com/photos/asinan/tags/ncdcftplisting/. The screen shots are stamped with US/Eastern times at which they were taken.

— Sinan

Steve McIntyre

Did anyone happen to save the deleted NOAA file?

• Sinan Unur
  Posted Nov 21, 2008 at 9:57 AM | Permalink

  Re: Steve McIntyre (#14), which one?

  I have anom-grid2-1880-current.dat.gz and grid_mean_temp_1880_current.dat.gz but not grid_1880_2008.dat.gz

  — Sinan

Jeff Alberts

Bravo for spelling “Viet Nam” correctly. A Viet Namese friend once told me that all words in the Viet Namese language were one syllable, so it seemed odd to me that the country name would be two syllables.

OT I know, sorry.

Steve McIntyre

grid_1880_2008.dat.gz was the current file that I’d like.

I modified by scripts only slightly to check out all the irrelevancies thrown at us by Deep Climate and ended up with one of his irrelevant files in my working session while NOAA was shredding the data. It looks like the same thing happened to you.

• Sinan Unur
  Posted Nov 21, 2008 at 10:08 AM | Permalink

  Re: Steve McIntyre (#18), sorry. I was haphazardly extracting, dumping, importing in a temporary directory until I realized that the contents of the directory changed. By that time, the files was already gone.

  I should have been more careful.

  — Sinan

Bob B

You think it’s cold now?–Accuweather calls out NOAA:

http://www.accuweather.com/mt-news-blogs.asp?partner=accuweather&blog=Meteomadness&pgurl=/mtweb/content/Meteomadness/archives/2008/11/you_think_its_cold_now.asp

StuartR

re Steve McIntyre #18

I just got back from work and checked and yes, grid_1880_2008.dat.gz was the very file I chose to download out of noodling curiosity at 23.46 GMT last night, do you still need it Steve?

crosspatch

DC, I don’t believe there is any “conflict” except when people get defensive when questions are asked about their data. The culture within the various US government agencies seems to be one of covering their backsides more than one of trying to get the data correct and often the attitude toward anyone who would question them is to first dismiss them as if they don’t know what they are talking about, and then to eventually change the data without any explanation. They seem to have a sort of barricade mentality when it comes to answering questions about their data and methods. Basically they treat people who find errors as “the enemy” rather than the resource that it is to getting things right.

Maybe some of this stems from public positions these agencies have taken where they have expressed various conclusions as certainties and then they use these data to back those conclusions. When the data is discovered to be incorrect, it calls the credibility of those conclusions and the people reaching them into question and that seems to elicit a defensive response rather than a partnering response to get to the bottom of what is really going on. There wouldn’t be so much appearance of “conflict” if the various government agencies didn’t put so much energy into defending their position rather than trying to learn what the real position should be. There is only one “side” and that is the side of what is really going on, what temperatures really are, etc. The data should be able to speak for itself provided it is accurate. What seems to keep turning up, though, are conclusions based on a combination of incorrect data, incorrect methods applied to the data, “missing” data that isn’t really missing but being kept out of the record for some reason, etc.

Steve Huntwork

I joined the U.S. Army in 1974 and I understood EXACTLY what you were trying to say! Of course, I eventually became an Army Meteorologist and specialized in satellite remote sensing.

Some of us “old farts” still remember when scientific studies were based upon reality, instead of computer models.

WAY TO GO!

Deep Climate

Redemption?

ftp://ftp.ncdc.noaa.gov/pub/data/er-ghcn-sst/

From the README

http://ftp.ncdc.noaa.gov/pub/data/er-ghcn-sst directory contains the global gridded dataset that has been compiled from GHCN (gridded using the “anomaly” method), and Sea Surface Temperatures, and merged using reconstruction techniques similar to those of the Extended Reconstructed Sea Surface Temperature dataset.

Took a quick look – has blended data to Oct. 2008.

• RomanM
  Posted Nov 21, 2008 at 3:19 PM | Permalink

  Re: Deep Climate (#33),

  The directory contains three data files, two of them with time stamps in 2007 and the third (temp5d6c.v2.adj.1880.last.dat.gz) time stamped today at 3:33 pm – presumably EST. Was it available yesterday?

Deep Climate

Yes, I noticed it was timestamped Nov. 21 (both file and archive). Basically I just went through the directories in pub/data until I found a likely sounding directory with a November timestamp. But I only did that today, so I can’t say for sure what was there yesterday.

A lot of people were challenging me to find the “right” data set, since I was so sure the other one was the “wrong” one (still am). Don’t know for certain that this is it, but my hunch is it will give a much closer match with the NCDC anomaly map.

Maurizio Morabito

I am not sure if this cavalier attitude with files and directories is the kind of scientific anarchism Feyerabend had in mind…

There must still be many people not understanding that to “put data into a publicly-available directory or website” is perfectly equivalent to “publishing” that same data (and to “announcing” them)

John Lang

All these errors, changing important data files mid-stream, and all the questionable “adjustments” to the temperature data to date are extremely important.

The 1880 to 2008 temperature trend puts us on a certain global warming track. It is much less right now than the global warming models predict but the trends to date show it might be a problem.

A little less error here and a little less “adjustment” there and a little less data file error there might put us on a completely different temperature trend track which indicates global warming is not that much of problem at all. A small change here or there might mean the difference between building a few more needed coal-fired power plants or closing all the existing ones.

The satellite data which seems to be less susceptible to these problems indicates that global warming is much, much less of an issue than the NOAA temp trends/errors and the GISS models indicate.

I, for one, think it is important these errors do not continue happening.

Deep Climate

#36:
Yes, this is the blended dataset, and I surmise it corresponds to the right hand graphic in the October, 2008 NCDC update. But at least it looks like it matches and has the right description and so on. Plus, you can get to it from the NCDC October update by following the links. So at least this part of the archive seems to be up to date.

As far as I can see an up to date “land only” grid cell dataset is not to be found. I have to presume it was/is supposed to be in ftp://ftp.ncdc.noaa.gov/pub/data/ghcn/v2/grid/

Now there were three different temp mean grid files in that directory (only one left, while another is gone, but can be found elsewehere). The readme definitely says that the anom-grid-1880-current.dat.gz is constructed with “anomaly” method, while another NCDIC access page points to grid_1880-2008_RVose.dat. Of course, neither of those datasets are up to date.

So even if you had a correct version of grid_1880_2008.dat, it’s not clear to me that it is/was the dataset used to generate the land only “anomaly method” chart. That’s what I mean by the “wrong” dataset. But who knows for sure? Isn’t there someone you could ask nicely, like one of the contacts listed – or is it too late for that?

The only thing that is clear that the archiving system is not, um, robust and the documentation is not up to date or consistent. I get the sense that NCDC is not too motivated to fix it very fast – “catching flies with vinegar” and all that.

• KimberleyCornish
  Posted Nov 21, 2008 at 10:18 PM | Permalink

  Re: Deep Climate (#41), One can reasonably take offence at the complete lack of humility in the tone of DC’s commentary. He/she has come into the discussion roaring like a lion about “errors” and (if he/she understands the concpt of decency) owes Steve a rather abject and very overdue apology. The apology ought to be stated ax such and delivered unambiguously in some form such as: “I was wrong and regret what I wrote. I will take on board your position in future and treat it with a good deal more respect than I have so far. I see that your auditing efforts are in fact fully justified and deserve my support. I will adopt this policy on my own blog.”

  • Nicolas J.
    Posted Nov 22, 2008 at 8:07 AM | Permalink

    Re: KimberleyCornish (#45),

    Rubbish. This is not an apology, this is a pledge to a sect guru.

    • kim
      Posted Nov 22, 2008 at 8:23 AM | Permalink

      Re: Nicolas J. (#52),

      I don’t know, NJ; the by-laws to that sect seem pretty arcane and subversive. Seems more like a cult to me.
      ==============================================================

Graeme Rodaughan

NOAA Claims “NOAA understands and predicts changes in the Earth’s environment, from the depths of the ocean to the surface of the sun, and conserves and manages our coastal and marine resources”.

However I would be far happier if they claimed “NOAA understands effective Data and Configuration Management techniques, and are able to provide 24 hour public access to both current and archived baselines of your data and any software that has been used to process it in accordance with “…pick any professional Industrial standard”.

Why “your data” – because the general public has paid for it’s gathering and storage – and if is being used to motivate policies that impact the lives of the general public. Both in themselves are sufficient reasons for an effective data and configuration management system to be in place.

Perhaps NOAA employees could spend some time at http://www.cmcrossroads.com/.

Steve McIntyre

#44. It might well be. However, I sent a polite email to Russell Vose notifying him of the defects and, in his shoes, I’d have sent back a short email saying so. Vose said that the problem lay with some other dude in NOAA and he’d inquire about it. It was the nature of the error that was odd and spoke to some sort of programming error in one of their modules.

It’s not like I think that any of this is necessarily a big issue. I’ve noted that there seems to be a clean version somewhere (as the graphic looks like it came from a clean version).

Because I’d raised the issue with them in a polite way, you’d also think that they’d alert me to the fact that they were withdrawing the data set to fix it. Much of the discussion here has arisen from the peculiarity of seeing data sets disappear while various folks were online looking at it disappear. This has happened on other occasions with other data sets and it’s always a little startling when people are online chatting about the data set and changes are taking place under our nose. We’ve seen this before with Hansen and with Mann.

“Deep Climate” also enlivened the discussion by charging me with gross errors, and all of his charges have proved unfounded.

Mike C

Steve Mc, 46 You are on the point about NOAA responsiveness. Unless you are Anthony Watts and have their thug in a thissle, you are as high a priority as crumbs on the coffee room floor. As for the DC poster, well, I had the time to read through the posts and that all became apparent.

JimB

“There is absolutely no reason whatsoever for data set updates to be as clumsy as evidenced in the recent failures. Don’t forget that company accounting and email archives must be kept accurate through periodic updates or someone can go to jail (Sarbox…) and often with complete rollback to any version.”

The problem is that there’s no “feedback” loop. It’s an open-ended system. In business, if you fail to keep things archived and retreivable in an orderly fashion, you face still penalties or jail. Your compliance is mandatory, and people from the board of directors to senior management sign off on reports stating compliance.

Here, there is no compliance, there are no penalties (direct, of course). That we apply these laws to businesses but not our own government is another story entirely.

JimB

Arthur Glass

‘Arrogant – from the latin, “a rogare” – “not asking”‘

Not exactly. The adjective is derived from the stem of present participle of the verb ‘adrogo.’ Now, the primitive meaning of the simple verb ‘rogo’ is indeed ‘ask’, but the verb developed a specifically political meaning, to ‘ask for’ or ‘propose’ a law. The ‘ad’ part is the prefix/preposition ‘ad’, meaning ‘to’ or ‘toward’, and the relevant meaning of the compound verb is ‘take to oneself, claim, assume’, with the connotation that the ‘taking’ is arbitrary and therefore not valid or justifiable.

Deep Climate

#44:
Well, sure this part:

“Steve, I was wrong and regret what I wrote.”

And I mean it. I can’t quite sign on to the rest of it though.

As far as NCDC documentation and archiving of the particular data goes, there are certainly problems. But I’m not sure as to what service level is mandated, or should be, for this kind of research data, or rectification of it. I don’t know, for example, if the out of date documentation in this particular directory has been pointed out to NCDC, even now.

It could be that removal of the two files is the first step in dealing with the immediate problem of the wonky datasets. There is a some sort of maintenance activity scheduled for today I belive, and I expect we’ll see an update next week.

Myself, I have an interest in the blended land-sea, zonal monthly anomaly data. Some of that is up to date and some is not. I wish it were all up to date, but I can’t get too exercised that it’s not (unless they don’t fix it within a reasonable amount of time).

You have to look at the whole context, too. Of course, I’ll take Steve’s word that he sent a polite email (or he may have published it somewhere, anyway). But the heading of this post (and presumably the video, which I haven’t watched) could be construed as, um, somewhat insulting. So … I expect the NCDC to take care of it, but not to be particularly friendly, informative or speedy.

Now a more substantive question. After all the brouhaha about the GHCN “copying” errors in the monthly data, I’m actually more interested in the general state of the GHCN monthly data than the NCDC temp analysis as such. Is there going to be a post on that in particular? Or can someone direct me to the right post?

crosspatch

“I’m actually more interested in the general state of the GHCN monthly data”

Yes, I believe that is what this whole thing has really called into question. How many times in the past have stations had monthly data carried over to a subsequent month but the difference wasn’t glaring enough to attract immediate attention. Until that question has been answered (and corrected), the monthly data can not be relied upon as being accurate. They are going to need to get to the bottom of whatever caused that carryover of data, fix it, and rebuild the monthly data files if those data are to be considered accurate. If they would also post the code that does that, the community of parties interested in those data may then also help spot errors and actually help in the maintenance of the code itself. It’s one of the benefits of “open source” code.

crosspatch

“I don’t think it’s necessary to have the code to do some useful checking.”

DC, you have apparently been here for about a week. SteveM and the others on this site have been doing this for a considerable period of time though the focus does shift from time to time to one thing or another. I don’t want to discourage you from making a contribution but you might want to keep in mind that the ftp sites involved are probably being scoured for relevant data by a dozen or more people at this point. The obvious places one might find by looking at a README file have likely been visited already.

‘I think taking a subset of GHCN station daily data for say 2008, generating averages for each month and comparing that to the published data would at least reveal the current scope of the “carryover” problem.’

How so? There might be only one such error one month, none the next month, and dozens of them another month. And 2007 might be completely different than 2008. In other words, the carryover problem doesn’t seem to be consistent. Because one happened at one month at one station, there is nothing that would indicate that the same station the same month would have the carryover problem in another year. Or that 2008 results would have any relevance to 2007 or that first half of 2008 bears any relationship to the second half of 2008 though there might be a pattern to what triggers these errors (I suspect it is some “magic” combination of missing values somewhere but that is pure speculation). All one could say from a result of the exercise you suggest is that so far in 2008 there have been X number of these errors.

SteveM has already stated that he has found at least one similar error in a spring reading that results in a month being much colder than it should be. These errors might be sprinkled at random throughout the record. We already know they are there, that means the monthly record is incorrect. How incorrect it is seems irrelevant at this time since finding the source of the problem, fixing it, and regenerating the monthly data will show exactly how wrong it was.

So it appears SteveM et al have discovered what appear to be two different problems. One was a monthly value being carried over to a subsequent month and a second problem where the set of data posted on the archive site is apparently not the data used to generate the published results. When SteveM brought the second problem to their attention, the relevant data began to disappear from the archive site without any explanation or ETA on when correct data would appear. This must be very frustrating for the researcher, particularly when different versions of data appear and disappear in rapid succession. It forces one to start over again from the beginning only to have the rug pulled out again when the data is changed again. But see, DC, these are the kinds of issues that have been typical whenever a problem has been discovered. Antics that would be quite unacceptable from a citizen or private company are de rigeur for these agencies. Could you imagine if an auto company were required to archive emissions control data for the EPA and engaged in such antics when an error is discovered? They would be fined or possibly prosecuted. This is particularly irritating because many of us work in occupations that do require rigid standards for data handling / archiving and the blatant deletion of incorrect data would be grounds for dismissal from most private entities yet is “business as usual” for our government agencies, it would seem. It is frustration more than conflict, I think.

• Sinan Unur
  Posted Nov 22, 2008 at 7:02 PM | Permalink

  Re: crosspatch (#60), thank you. That is an excellent summary.

  — Sinan

Hu McCulloch

Re RomanM, #22,
Thanks for the explanation of NOAA’s “first difference method”. Your post made a lot more sense than “Deep Climate”‘s citation of the definition of first differencing at #99 on the “Anomalous in Finland” thread.

As described, the “First Difference Method” makes sense when not all stations cover the whole time period. However, when a single observation is missing for one station, it appears to treat the resumed data as if it were a new station, and therefore loses the valuable information that this is in fact the same station.

So apparently it’s just as well if NOAA doesn’t really use it, as now claimed by DC.

• Sinan Unur
  Posted Nov 23, 2008 at 7:40 AM | Permalink

  Re: Hu McCulloch (#62), their so-called first-differencing method is more like a 12-differencing method given that the time unit in consideration is a month.

  I am not a fan of the anomaly method (especially the freedom it affords its users in picking the ‘normal’ period).

  Just two minor observations.

  — Sinan

Deep Climate

#62:
The NOAA discussion of transition from FD to anomaly method is here:
http://www.ncdc.noaa.gov/oa/climate/research/ghcn/ghcngrid.html

#60:
I don’t really want to get into a discussion about “antics” and so forth, although I know your POV is fairly common here.

As far as having something useful to contribute, I’ll leave it to others to judge:
http://www.climateaudit.org/?p=4370#comment-313046 on GHCN monthly.

Craig Loehle

“Hello Viet Nam” just came on my local cable channel–how weird is that?

Mike Bryant

hey Mike C.,
Mabe someone needs to go knock on their door.

Steve McIntyre

A new version of ftp://ftp.ncdc.noaa.gov/pub/data/ghcn/v2/grid/grid_1880_2008.dat.gz was posted on the NOAA website today without explanation or apology.

PS. It appears to be the same version as the one that was deleted last week. Gridcell 329 is the same as before and is still problematic.

Deep Climate

It’s December now – so this is the first to have November, 2008 data. It would normally be updated at least once a month at least to incorporate the latest month’s data.