"Anomalous" in Finland

Occasionally, I’ve been criticized for spending too much time on NASA GISS. Since we aim to please even our severest critics, let’s spend a little time today on NOAA, which has just reported the second warmest October of all time.

I downloaded the NOAA gridded data (who mercifully, in this case, at least use .gz instead of the medieval .Z compression so beloved of antique climate scientists). I’ve uploaded the read script and you can obtain the data into R as follows in 2592 columns (Jones grid order) starting in 1880:

source(“http://data.climateaudit.org/scripts/gridcell/collation.functions.txt”)
noaa=download.noaa(); noaa=noaa/100; dimnames(noaa)[[2]]=1:2592
dim(noaa)

The first thing that most data analysts do with fresh data is to look for extreme cases. Here I picked out gridcells with October 2008 anomalies over 5 deg C (10 of them).

N=nrow(noaa); x=noaa[N-2,];
temp=(x>5.00)&!is.na(x); id=as.numeric(names(x[temp]))
x[temp]
# 196 208 221 303 329 418 421 485 634 710
#6.49 5.24 5.29 5.45 7.42 5.20 5.22 5.23 5.70 6.24

I then identified the GHCN stations that were in these gridcells, noting in particular, stations with values in 2007 and 2008. Relevant stations were Ostrov Vize, Ostrov Kotel, Barrow, Cambridge Bay, Karuesuando/Haparanda, Ergobasen, Viljujsk, Koplasevo, Dauunmod and Yanji, a couple of which we encountered in connection with GISS.

download.file(“http://data.climateaudit.org/data/giss/giss.info.tab”,”temp.dat”,mode=”wb”);load(“temp.dat”)
giss.info$jones=jones(giss.info$lat,giss.info$long)
test=giss.info[ !is.na(match(giss.info$jones,id)),c(1:6,22,29)]
order1=order(test$jones,test$id)
test[order1,]

Some regular CA contributors are from Finland. Combined with the fact that the October 2008 was the largest in the entire world, this seemed like a good one to analyze. I downloaded the most recent GISS dset1 versions of Haparanda and Karuesuando and compared them to the corresponding NOAA gridcell. For each GISS dset1 series (which are expressed in deg C), I converted them to anomalies prior to the comparison.

g= function(A) ts( c(t(A[[1]][,2:13])),start=c(A[[1]][1,1],1),freq=12)
station1=anom(g( read.giss(giss.info$id[6848],dset=1)));
station2=anom(g(read.giss(giss.info$id[6852],dset=1)));#Bor
ts.union(station1,station2,noaa[,329])

The October 2008 anomalies for Haparanda and Karuesuando were garden variety – nothing that yielded the NOAA anomaly:

Hap. Kar. NOAA
Aug 2008 -1.04 -0.97 0.64
Sep 2008 -0.42 -0.67 -2.49
Oct 2008 0.07 1.01 7.42

A little puzzled by this, I printed out the NOAA history for gridcell 329, excerpts are shown below:
1971 2.57 -0.78 -4.85 -1.49 -0.85 0.15 -0.94 NA NA NA NA NA
1972 2.92 1.10 1.85 0.36 -0.75 2.37 3.21 NA NA NA NA NA
1973 7.47 0.90 2.30 -0.12 -0.62 1.22 3.96 NA NA NA NA NA
1974 5.72 5.45 2.55 1.83 0.10 1.60 0.46 NA NA NA NA NA
1975 3.19 6.40 4.90 0.31 1.15 -2.20 -2.02 NA NA NA NA NA

….
1999 -1.76 -0.68 1.92 2.06 -1.10 1.82 0.31 -0.01 0.41 8.62 7.09 -9.53
2000 4.14 3.32 2.12 1.06 1.30 -0.33 1.31 2.09 -1.14 11.12 7.14 -5.53
2001 6.54 -1.03 -2.78 0.56 0.10 1.92 0.36 2.39 0.46 7.67 2.29 -6.63
2002 0.89 4.22 1.42 3.41 2.85 2.97 1.76 4.59 -2.09 3.87 -2.16 -9.78
2003 -4.51 5.77 4.72 0.91 1.65 0.02 4.01 2.59 -1.84 6.87 4.19 -5.78
2004 1.79 0.72 2.42 2.21 1.25 -0.93 1.61 2.64 -0.49 7.17 0.99 -2.33
2005 6.54 2.62 -0.73 2.36 -0.80 0.92 2.21 3.24 -0.99 9.22 6.59 -6.68
2006 4.24 0.32 -3.33 2.81 1.90 0.67 0.41 5.19 -0.79 5.57 4.19 -0.58
2007 0.94 -3.58 5.18 2.11 0.45 0.72 0.51 2.84 -2.24 9.87 3.14 0.57
2008 6.14 4.97 0.32 0.81 0.15 -0.63 -0.04 0.64 -2.49 7.42 NA NA

So the October 2008 anomaly was very high to say the least, but the NOAA anomaly is always high in October. It was 11.12 deg C in 2000. WTF is going on? And what happens prior to 1991. NOAA has gridded values from Jan to July, but not from August to December during the entire period from 1880 to 1991.

Here’s something odd. I calculated my own anomaly on the GISS/GHCN dset1 versions of Haparanda and Karasuando and took a simple average. I compared this to the NOAA gridded version. For every October from 1992 to 2008, the NOAA series was exactly 6.88 deg C higher than the average and for every December it was 7.53 deg C lower. Over the 5 months with oddball data, the NOAA gridcell averaged 0.2 deg C higher than the average of the two stations.

2002 -0.23 -0.24 -0.19 -0.11 -0.06 0.00 0.00 1.64 -1.94 6.88 1.96 -7.52
2003 -0.23 -0.24 -0.19 -0.11 -0.06 0.00 0.00 1.65 -1.95 6.88 1.97 -7.53
2004 -0.23 -0.24 -0.19 -0.11 -0.05 0.00 0.00 1.65 -1.95 6.88 1.97 -7.53
2005 -0.24 -0.24 -0.19 -0.11 -0.06 0.00 0.00 1.65 -1.94 6.88 1.96 -7.52
2006 -0.23 -0.24 -0.19 -0.11 -0.06 0.00 -0.01 1.65 -1.95 6.88 1.97 -7.53

The comparison also tells us something else – because the match was so precise even if weird, it shows that the NOAA gridcell was calculated from the Haparanda and Karasuando stations. My surmise is that something is wrong with how they calculated their normals, but for sure, there’s something wrong.

Update – 10 pm: see comment below comparing NOAA plot and emulated NOAA plot. It looks like the version of the data used for the NOAA October graphic does not contain the error observed here. It appears that the archived data (See link in comment below) differs from the data used in the NOAA October graphic and that I stumbled upon one of the discrepant gridcells. I’ve emulated most parts of the NOAA graphic quite accurately with the northern Finland gridcell sticking out quite dramatically. There are a couple of other interesting discrepancies, which readers can note for themselves.

138 Comments

  1. Matt
    Posted Nov 19, 2008 at 6:51 PM | Permalink

    Wow, no wonder NOAA and GISS don’t want you to know what’s really up their sleeves. It would be one thing if it was HARD to find things that make no sense…

  2. Steve McIntyre
    Posted Nov 19, 2008 at 6:54 PM | Permalink

    I just looked at Ergobacen and the NOAA anomaly always deviates from my calculation for the same amount in each month form 1991 on, but nothing glaring like Finland. The size of the Finnish problem probably has something to do with losing the pre-1990 data. Some of these things are brainteasers, that’s for sure.

  3. Clark
    Posted Nov 19, 2008 at 6:57 PM | Permalink

    Doesn’t anyone even look at this data? You would think anyone with a couple brain cell working would see the Sept, Oct, Nov Dec anomalies year after year and realize there is a huge problem with the algorithm.

    Too bad NOAA won’t release their code so some semblance of sanity could be imposed on the data.

  4. Jeff Alberts
    Posted Nov 19, 2008 at 7:24 PM | Permalink

    which has just reported the second warmest October of all time.

    Which isn’t even true or knowable.

  5. Jeff Alberts
    Posted Nov 19, 2008 at 7:26 PM | Permalink

    Combined with the fact that the October 2008 was the largest in the entire world, this seemed like a good one to analyze.

    Seems to be something missing from that sentence…

  6. Mike C
    Posted Nov 19, 2008 at 7:27 PM | Permalink

    They always start off with this statement:
    “Based on preliminary data,” then they go on with the dramatic claims about how the Earth is boiling as is the case this month, “the globally averaged combined land and sea surface temperature was the second warmest on record for October and ninth warmest on record for the January-October year-to-date period.” What surprised me this month is that “based on preliminary data” their analysis that October this year was the warmest land record did not make it into the first paragraph. Perhaps they were watching the blogs for the last week (and figured no one would buy it)?

    If you go back through their monthly reports they are basically all the same. Then the preliminary data gets changed but without a peep. Remember when 2006 was the warmest on record? Yep, based on preliminary data. They get all of the press from the official doom and gloom statements then quietly change them at a different time.

  7. Joel
    Posted Nov 19, 2008 at 7:48 PM | Permalink

    Hey Steve,
    There is an “anomaly” in your fourth paragraph from the end of this entry:

    I think you meant to say “WTF is going on?” rather than “WFT is going on?” – unless this is a Canadian dialect that I am not familiar with;)

  8. Steve McIntyre
    Posted Nov 19, 2008 at 7:54 PM | Permalink

    For another red October station (Yanji), here are the differences between GISS dset1 and the NOAA gridcell anomaly for a few recent years:

    2002 6.58 -1.70 1.96 -1.24 0.98 -0.05 -0.54 -0.02 0.49 3.61 -0.64 3.74
    2003 6.58 -1.70 1.96 -1.24 0.98 -0.05 -0.54 -0.02 0.49 3.61 -0.64 3.74
    2004 NA -1.70 1.96 -1.24 0.98 -0.05 -0.54 -0.02 0.49 3.61 -0.64 3.74
    2005 NA -1.70 1.96 -1.24 0.98 -0.05 -0.54 -0.02 0.49 3.61 -0.64 2.21
    2006 6.31 -1.70 2.21 -1.24 0.53 1.52 -0.54 0.83 0.59 4.66 0.06 3.41
    2007 6.31 -1.70 2.78 -1.24 1.63 0.40 0.06 0.31 0.79 3.59 -0.57 2.61
    2008 6.31 -1.70 2.21 -1.81 2.03 1.05 -0.36 0.98 0.79 3.59 NA NA
    >

    • Gerald Machnee
      Posted Nov 19, 2008 at 8:33 PM | Permalink

      Re: Steve McIntyre (#8),
      No wonder the glaciers are melting!

    • David L Hagen
      Posted Nov 19, 2008 at 10:38 PM | Permalink

      Re: Steve McIntyre (#8),
      Looks like “Indian Summer” in Paul Bunyan’s logging camp in Minnesnowta. At least there are alot of Finns in northern Minnesota. Could this be starting up the saunas?

  9. MattN
    Posted Nov 19, 2008 at 8:11 PM | Permalink

    Holy CRAP! What in the hell is going on over there at NOAA?!?!? I can tell you with much certainty that the possability of the last 10 Octobers in a row being THAT warm in one location is near zero.
    snip

  10. Mike Bryant
    Posted Nov 19, 2008 at 8:12 PM | Permalink

    The aberration in these anomalies seems quite abnormal. These departures are a significant deviation from the satellite data. The eccentricities and inconsistencies of both NOAA and GISS require that the irregularities, oddities and peculiarities are explained forthwith.

    Fun with the thesaurus. 🙂

  11. jae
    Posted Nov 19, 2008 at 8:23 PM | Permalink

    Steve is getting to the bottom of this nonsense, slowly but surely. It is so entertaining, for a person who has worked for the government (albeit briefly) and knows how lax it is!

  12. Gerald Machnee
    Posted Nov 19, 2008 at 8:30 PM | Permalink

    Something is rotten in the state of Finland.
    They will have it fixed in 24 hours.

  13. steven mosher
    Posted Nov 19, 2008 at 8:31 PM | Permalink

    Perhaps he data is post “fill in missing data where possible” step” I might be missing something. Prior to 1991, the aug-dec
    months are missing. Perhaps after that you get sparse readings (sept, nov) and the “fill in missing data” process kicks in occassionally and has “undesired” side effects.

  14. Dan K
    Posted Nov 19, 2008 at 8:36 PM | Permalink

    You know, it would really help us laymen if someone translated all of these articles into English. Having received a C- in my first semester of calculus and later ending up as a mere lawyer, it feels like I’m reading a … a…. words simply fail me. I understand that all is not right with the world, but can’t articulate what information the scientific techno-speak intends to convey.

  15. Steve McIntyre
    Posted Nov 19, 2008 at 8:36 PM | Permalink

    #13. The data is not really “missing”. It’s only MIN – “missing in NOAA”: for example:

    Hap Kar NOAA
    Jan 1981 -0.65 2.00 0.84
    Feb 1981 -3.10 -1.38 -2.13
    Mar 1981 -5.85 -5.63 -5.70
    Apr 1981 0.28 0.46 0.41
    May 1981 1.25 -0.74 0.33
    Jun 1981 -3.18 -2.78 -2.95
    Jul 1981 0.06 -0.33 -0.09
    Aug 1981 -0.54 -1.17 NA
    Sep 1981 0.18 -0.57 NA
    Oct 1981 0.27 0.31 NA
    Nov 1981 -1.20 0.15 NA
    Dec 1981 -7.24 -6.37 NA
    Jan 1982 -0.85 -4.50 -2.48
    Feb 1982 4.70 3.62 4.30
    ….

  16. Posted Nov 19, 2008 at 8:38 PM | Permalink

    The 11/17 NOAA/NCDC statement that October 2008 had the warmest land temperatures in 129 years is still online at http://www.ncdc.noaa.gov/oa/climate/research/2008/oct/global.html.

  17. Steve McIntyre
    Posted Nov 19, 2008 at 8:40 PM | Permalink

    #15. Just skip over the programming lines. I put them there in case people want to confirm the results for themselves. I would really like it if other people would do this – IMO, it’s the modern equivalent of including equations in articles and makes it possible to understand what people are doing. Too much climate science is written with neither equations nor programming and becomes incomprehensible not because the topic is inaccessible, but because the form of exposition is unclear.

  18. Posted Nov 19, 2008 at 8:49 PM | Permalink

    The cool dot along the Mexico/Texas border is a bit puzzling. The US side of the border was anomalously cool by 0.5C, at most. If the US side was 50% of the cell land area then the Mexican side had to be cool by 8C or so to average out to -4C. Brrr.

  19. Alan S. Blue
    Posted Nov 19, 2008 at 8:54 PM | Permalink

    Why don’t these immediately run into issues with a cross-check versus, say, the RSS data?

    I mean, I assume it is “We just don’t do that.” But wouldn’t this seem like an excellent outlier-detector? They’re apples-vs-pears, of course, but an RSS-derived average gridcell temperature versus any of the ground based techniques should at least point out egregious examples, yes?

    IOW: What does RSS say about the area near Yanji or Ergobacen? And can the RSS data be forced through the “gridcell” shenanigans to get a gridcell average?

  20. Steve McIntyre
    Posted Nov 19, 2008 at 9:24 PM | Permalink

    Hu, I tried to emulate the NOAA diagram shown below. Following it is my emulation (not exact but pretty close as to method) based on the data in the file ftp://ftp.ncdc.noaa.gov/pub/data/ghcn/v2/grid/grid_1880_2008.dat.gz downloaded today.

    There’s a very large discrepancy between the two plots in the very gridcell in question. It appears that there are different versions of the NOAA data floating around and the version used in their plot does NOT contain the screwed up normals in the version presently archived.


  21. Jeff C.
    Posted Nov 19, 2008 at 9:25 PM | Permalink

    Looking forward to the argument that since December is always off by a slighty larger magnitude with the opposite sign, worst-case it’s a wash. It doesn’t matter that the algorithm is suspect and has apparently never been fully tested. We’re talking big-picture narrative here.

    The idea of a gridcell sanity check against RSS/UAH is interesting. I have never seen the satellite data broken down in a format that would be compatible. Anyone know if it is available?

    • braddles
      Posted Nov 19, 2008 at 11:25 PM | Permalink

      Re: Jeff C. (#23), RSS put up monthly maps of satellite temps here. They use a different map projection, and a different base period to the surface people, but it useful for eyeball comparison.

  22. Mike C
    Posted Nov 19, 2008 at 9:26 PM | Permalink

    Hu 17… that’s what I was talking about, once they publish it, they do not correct it.

  23. Ross McKitrick
    Posted Nov 19, 2008 at 10:06 PM | Permalink

    #23: The UAH data are published on 2.5 degree grid cells. Over land it would be possible to take an interval, compute the trend in the surface and MSU series, then see if the differences are merely random or correlate to indicators of inhomogeneities, data quality and improper sampling conditions. Wouldn’t that be an interesting exercize.

    • Sam Urbinto
      Posted Nov 20, 2008 at 11:09 AM | Permalink

      Re: Ross McKitrick (#25),

      Paper, data, summary, background, issues.

      If only everyone involved in this subject had a page like that for their articles!

  24. Jeff C.
    Posted Nov 19, 2008 at 10:39 PM | Permalink

    Dr. McKitrick – thanks for the reply. I’ve heard of the paper you wrote with Dr. Michaels, but am embarrassed to admit I’ve never read it. Just got a copy and it should keep me busy for a few hours. Thanks for the tip.

  25. Jeremy
    Posted Nov 19, 2008 at 11:43 PM | Permalink

    What Language is that you’re using Steve? It’s been a while since I programmed in anything other than Matlab, Mathematica, or Labview.

  26. Steve McIntyre
    Posted Nov 20, 2008 at 12:16 AM | Permalink

    R

  27. Kalle
    Posted Nov 20, 2008 at 12:23 AM | Permalink

    Haparanda and Karuesuando are situated very close to the Finnish border but in Sweden, not Finland.

  28. Kalle
    Posted Nov 20, 2008 at 12:26 AM | Permalink

    Karesuando not Karuesuando, sorry.

  29. Sven
    Posted Nov 20, 2008 at 12:33 AM | Permalink

    Sorry for cross posting but the HadCru thread seems to be dead already…

    Can someone enlighten me on how the HADCRUT yearly anomaly figures are calculated. The most logical thing would seem to be to just take a simple average of monthly figures by adding the monthly anomalies together and dividing the result by 12. I started looking at the years that are similar to 2008 and found out that (apart from 1995) there are significant differences between the HADCRUT yearly figure and the average of monthly anomalies. For example (the first figure being the HADCRUT yearly and the second the average of months):
    1990 – 0.248, 0.254
    1995 – 0.276, 0.276
    1997 – 0.355, 0.350
    1999 – 0.262, 0.296
    2000 – 0.238, 0.270
    2008(10 months) – 0.304, 0.315
    The largest difference is somehow in the years 1999 and 2000. I wonder how would the higher anomalies for these years influence the trend line? Would there be a decline since 1997 already? Strange anyhow…

  30. tty
    Posted Nov 20, 2008 at 2:12 AM | Permalink

    Those data are utterly nonsensical. Haparanda and Karesuando are both in Sweden (though situated along the border river Torne älv). They are about 200 miles apart and have rather different climates (Haparanda is coastal, Karesuando is inland with a quite continental climate). I have checked the monthly anomalies published by SMHI (the Swedish weather service), these are published in map format and are available from 2005 on (http://www.smhi.se/cmp/jsp/polopoly.jsp?d=11490&l=sv), so I can only give approximate values. The October anomalies are:

    2005 H: + 2 degrees, K + 2 to 3 degrees
    2006 H: – 2 to 3 degrees, K – 1 to 2 degrees
    2007 H: + 2 to 3 degrees, K + 3 to 4 degrees
    2008 H: + 0 to 1 degree, K + 0 to 1 degree

    These are compared to the internationally agreed 1961-1990 standard period (not used by GISS!). Sweden has a good ground station net, with a high proportion of rural sites. Incidentally there is of course historical data available for both Karesuando and Haparanda. For Haparanda there are data going back to 1802, though the older part of the series is inhomogenous (parts of it is from Torneå, the now finnish town across the river).

  31. John Finn
    Posted Nov 20, 2008 at 4:04 AM | Permalink

    Re: #33

    Sven

    I’d always assumed the reason that annual anomalies were not exactly (monthly anomalies)/12 was due to the different number of days in each month, but some of the differences do look abit large for that to be the explanation.

  32. Sven
    Posted Nov 20, 2008 at 4:31 AM | Permalink

    It does look weird. I’ve calculated the averages for all the years from 1997 to 2008. Here are the differences (the first line is HadCrut3 annual, the second simple averages of HadCrut3 monthly anomalies):
    1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008
    .350 .546 .296 .270 .408 .464 .473 .447 .482 .422 .405 .315
    .355 .515 .262 .238 .400 .455 .457 .432 .479 .422 .403 .304

  33. Sven
    Posted Nov 20, 2008 at 5:00 AM | Permalink

    Thanks, UC

  34. Sune
    Posted Nov 20, 2008 at 6:01 AM | Permalink

    The US presumably has the best meteorological observational network on the planet. Looking at Steve’s anomaly plots, something strikes out I think and perhaps it has already been a topic here at CA.

    But the US is a mix of warming and cooling trends as one would expect if the climate varies about a stable average. However, nearly any other land mass only shows a warming trend. Why this discrepancy? Is global warming not affecting the US? Or would the warming trends of the rest of the world simply disappear if they had a network of US-type-quality?

    It seems as if global warming is not really global. The US need not worry I think.

    • Posted Nov 20, 2008 at 6:29 AM | Permalink

      Re: Sune (#39),

      However, nearly any other land mass only shows a warming trend.

      For example, Finland has quite similar history as US,

      (thin red Helsinki UHI corrected ). Of course CRU procedure reduces ’30s temperature a bit :

      ( picture from https://oa.doria.fi/handle/10024/2634 Tuomenvirta, Heikki; Doctoral dissertation )

  35. Chris
    Posted Nov 20, 2008 at 6:31 AM | Permalink

    The standard response is something like “well the US is only 2% of the globe’s surface area”, it’s just a localised anomaly.

    My response to this would be, yes but it’s a much greater % of the surface area for the NH at the latitudes it covers. And we all know about the global circulation patterns in the atmosphere see e.g. the animation here:
    http://www.gerhards.net/astro/wolken_200705_en.html

    So how can the warmed air circulating at those latitudes (over the warmed oceans as well as land) be failing to warm the US as expected each time it goes round the globe, so to speak? Yes regional factors can play a part, but I don’t think they fully explain the discrepancy, and I think it’s quite likely that global temperature rises have been closer to the (still exaggerated?) US figures than is assumed.

  36. Mike Bryant
    Posted Nov 20, 2008 at 7:47 AM | Permalink

    Chris,
    If you keep talking like that, people will think you are a skeptic.
    Mike

  37. Gerald Machnee
    Posted Nov 20, 2008 at 8:15 AM | Permalink

    Looking at those dots – sure looks like “Blue Hawaii” to me.

  38. Timo Hämeranta
    Posted Nov 20, 2008 at 8:58 AM | Permalink

    The following picture presents Sodankylä, Lappland, Finland October mean temperatures 1951-2008:

    Source: Finnish Meteorological Institute at
    http://www.fmi.fi/saa/tilastot_64.html#10

  39. Steve McIntyre
    Posted Nov 20, 2008 at 9:44 AM | Permalink

    Timo. if you (or anyone else) can post up links to actual (Finnish) digital data files for these stations, I’d appreciate it.

  40. Steve McIntyre
    Posted Nov 20, 2008 at 9:52 AM | Permalink

    I sent the following inquiry to Russell Vose of NOAA (including the graphics shown above):

    I downloaded the file ftp://ftp.ncdc.noaa.gov/pub/data/ghcn/v2/grid/grid_1880_2008.dat.gz and am puzzled by some of the information.

    Here is a plot at your website and my emulation of this plot using the downloaded data. Some nuances of the emulation (e.g. dot radii) don’t quite match, the overall match is close enough to show that I’m plotting the right data. The archived data has a very loud Oct 2008 anomaly in northern Finland-Sweden (7.42 deg C) see lower plot, which is not present in the reported graphic. I examined the underlying data in this gridcell (based on Haparanda-Karausando) and there is definitely a problem with the archived data set. One sees some other puzzling discrepancies e.g. South America.

    I also notice that this contains a few gridcells that do not appear to be present in the GHCN data. Is the NOAA gridded data based on the GHCN compilation or are there some variations? Regards, Steve McIntyre

  41. George T
    Posted Nov 20, 2008 at 11:02 AM | Permalink

    What is it with October data? Why the vast variance in anomalies?

    And do people in northern Finland realize that it is really 7 or 8 degree warmer than they think it is? Shouldn’t the Today Show be doing a remote from that endangered clime?

  42. Good Captain
    Posted Nov 20, 2008 at 11:28 AM | Permalink

    I know much has been made over the months and years of attempts to get a clean and clear look at AGW proponents’ data files and methodologies. I have read of numerous FOIA requests but I do not recall any references to searches w/in the national archives. It is my general understanding that all federal government records ultimately make their way there. Are there any readers familiar w/ the archiving process to confirm or elaborate?

  43. Sam Urbinto
    Posted Nov 20, 2008 at 11:57 AM | Permalink

    Here’s the climatology periods:

    GISS: 1951-1980
    WMO ‘official normal’, NOAA: 1961-1990
    FMI: 1971-2000

    ————————-

    Steve Said:

    Occasionally, I’ve been criticized for spending too much time on NASA GISS. Since we aim to please even our severest critics, let’s spend a little time today on NOAA

    Another question is if NOAA will thank those who help them make their product better or point out potential issues regardless if they turn out to be issues or not. Or if they will glare at the occasional critic and say to them “Thanks a lot! Why don’t you keep your mouth shut next time.”

    —————-

    FMI said:

    Finland’s Climate

    The main factor influencing Finland’s climate is the country’s geographical position between the 60th and 70th northern parallels in the Eurasian continent’s coastal zone, which shows characteristics of both a maritime and a continental climate, depending on the direction of air flow. The mean temperature in Finland is several degrees (as much as 10°C in winter) higher than that of other areas in these latitudes, e.g. Siberia and south Greenland. The temperature is raised by the Baltic Sea, inland waters and, above all, by airflows from the Atlantic, which are warmed by the Gulf Stream.

    When westerly winds prevail, the weather is warm and clear in most of the country due to the ‘föhn’ phenomenon caused by the Keel range. Despite the moderating effect of the ocean, the Asian continental climate also extends to Finland at times, manifesting itself as severe cold in winter and extreme heat in summer.

    Since Finland is located in the zone of prevailing westerlies where tropical and polar air masses meet, weather types can change quite rapidly, particularly in winter. The systems known to affect Finnish weather are the low-pressure system usually found near Iceland and the high-pressure systems in Siberia and the Azores. The position and strength of these systems vary, and any one of them can dominate the weather for a considerable time.

    According to Köppen’s climate classification, Finland belongs wholly to the temperate coniferous-mixed forest zone with cold, wet winters, where the mean temperature of the warmest month is no lower than 10°C and that of the coldest month no higher than -3°C, and where the rainfall is, on average, moderate in all seasons.

  44. STAFFAN LINDSTROEM
    Posted Nov 20, 2008 at 12:16 PM | Permalink

    #34 tty, using Tu Tiempo and SMHI “Dataserier för observationsstationer” , I would say Haparanda was 1-1.5C above normal
    Which was +2.1C 1961-1990…NOAA having Berlin 4 degrees C warmer is
    just ridiculous…Normal temp around 10C which it WAS…[+-0.5 deg]

  45. Steve McIntyre
    Posted Nov 20, 2008 at 12:21 PM | Permalink

    Folks, no need to belabor the point that climate in n Finland-Sweden was usual October. The problem is in the “normals” for this gridcell. Without finding out what NOAA did, there’s not much more that can be done right now.

  46. Stuart Harmon
    Posted Nov 20, 2008 at 12:54 PM | Permalink

    Steve

    The following link is to an article regarding the temperature in Spitsbergen in 1922.

    Click to access 1922%20SvalbardWarming%20MONTHLY%20WEATHER%20REVIEW%20.pdf

    The site in general has some interesting data regarding both good and difficult sea ice conditions in the Arctic and also temperature data elsewhere.

    No doubt you will be able to make sense of it. I hope it helps.

    Belated apologies for an earlier post on politics which you allowed and where I confused you as being from the USA.

  47. Lars Kamél
    Posted Nov 20, 2008 at 12:57 PM | Permalink

    I would like to point out that daily data for many European weather stations may be found here:
    http://eca.knmi.nl/
    Of course, it takes a lot of calculations to transfer these data to more convinient records, such as monthly and annual means. Maybe an R expert could do this rather easily? It would be interesting to compare the results of such calculations to the data used by NOAA and GISS.

  48. Stuart Harmon
    Posted Nov 20, 2008 at 1:05 PM | Permalink

    Steve

    My earlier post I forgot to leave this link:-
    http://www.climate4you.com/SvalbardTemperatureSince1912.htm

  49. Steve McIntyre
    Posted Nov 20, 2008 at 1:32 PM | Permalink

    #57. This sort of information is not nearly as helpful as it might be. What would be more helpful is a description of the steps that you did to obtain station data and the long url of a resulting data set.

    Also if you notice any details on provenance as sometimes these data sets simply draw on GHCN.

  50. Bill F
    Posted Nov 20, 2008 at 1:44 PM | Permalink

    #19 David Smith,

    If I am not mistaken, that cold spot on the Texas/Mexico border is related to an area that got pounded with rain for several weeks during September/October this year. They had record flooding in the mountains south of the border and along the Rio Grande near Presidio and Castellon. Strangely, very little of the precipitation made it across the border, leaving Southwest Texas with the hot dry air while Mexico had clouds and cooler rainy conditions.

  51. tty
    Posted Nov 20, 2008 at 2:16 PM | Permalink

    Re 45:

    Karesuando daily data 1961-2007:
    http://data.smhi.se/met/climate/time_series/day/temperature/SMHI_day_temperature_clim_19283.txt

    Haparanda daily data 1961-2007: http://data.smhi.se/met/climate/time_series/day/temperature/SMHI_day_temperature_clim_16395.txt

    Term description: http://data.smhi.se/met/climate/time_series/day/temperature/file_content_temperature.pdf

    Karesuando monthly: http://data.smhi.se/met/climate/time_series/month_year/precipitation_temperature/SMHI_month_year_precipitation_temperature_clim_19283.txt

    Haparanda monthly: http://data.smhi.se/met/climate/time_series/month_year/precipitation_temperature/SMHI_month_year_precipitation_temperature_clim_16395.txt

    Term description: http://data.smhi.se/met/climate/time_series/month_year/precipitation_temperature/file_content_month_year_precipitation_temperature.pdf

    There are 3-hourly data too, but I suppose you are less interested in those.

  52. Deep Climate
    Posted Nov 20, 2008 at 2:21 PM | Permalink

    Steve,
    Perhaps you can clear up a couple of things as the exposition is somewhat confusing.

    1) Does this boil down to a discrepancy between the apparently weird data you downloaded, and the actual published NCDC anomaly map? i.e. are you questioning the NCDC published anomalies, or only whatever it is you downloaded?

    2) I see a reference to download of GISS/GHCN data and conversion to anomalies. How did you do that exactly? (The base period is different for NCDC and GISS of course). Wouldn’t it be better to work directly from GHCN and emulate the NCDC analysis from there?

  53. tty
    Posted Nov 20, 2008 at 2:43 PM | Permalink

    Re 58

    Homogenizing those Svalbard series must have been difficult, particularly during WW 2 when there were several moves and abandonments of stations due to german military activity, and also the shift (in the mid 70’s I think) from Isfjord Radio to Longyearbyen Airport. Isfjord Radio is in a very exposed site at the mouth of Isfjorden while Longyeabyen Airport is deep inside the fjord system.

  54. Steve McIntyre
    Posted Nov 20, 2008 at 3:16 PM | Permalink

    #62. I don’t know how to diagnose the problem right now. Russell Vose acknowledged my email; he forwarded it to another NOAA employee who will hopefully respond.

    GHCN only supplies dset0 versions and I don’t know how NOAA grids the station data. I used GISS dset1 because it combines the various dset0 series from GHCN. Their method is a bit a wonky, but only a bit wonky and reconciles enough to dset0 data for the present analysis. Because of the exact monthly differences being replicated, you can tell that NOAA and GISS used the same versions in the period in question.

  55. Steve McIntyre
    Posted Nov 20, 2008 at 3:18 PM | Permalink

    #61. Excellent – that’s helpful. Do you have a URL showing a list of station identification codes as well?

  56. tty
    Posted Nov 20, 2008 at 3:52 PM | Permalink

    Re 65

    There is a list here:

    http://www.smhi.se/content/1/c6/02/25/37/attatchments/ps.txt

  57. merenrannalta
    Posted Nov 20, 2008 at 4:48 PM | Permalink

    This might be little off-topic, but just a reminder: we are now talking about basicly the same area, where Finnish dendroclimatologists have collected 7600 years worth of climate records. Oldest discoveries are from above treeline tree trunk fossils preserved in the cold lakes.

    Click to access gt_trace2008h.pdf

    http://lustiag.pp.fi/climate_change.htm

  58. David Smith
    Posted Nov 20, 2008 at 5:12 PM | Permalink

    Re #60 Interesting possibility, Bill.

    I checked the NOAA precipitation maps for Mexico and found that there were no heavy rains in the area of interest. There were heavy rains in north-central Mexico but that was south of the cell in question. The Hovmoeller surface temperature image for the cell shows a couple of cool spells during October but overall temperature appears to be about normal.

    I’ll look for Mexican station records this weekend.

  59. Deep Climate
    Posted Nov 20, 2008 at 5:27 PM | Permalink

    OK, let’s back up a little bit here.

    I was going to ask you exactly which files you downloaded (since you didn’t bother to say), but then I rooted through your R code and found this URL in the download.noaa() function:
    ftp://ftp.ncdc.noaa.gov/pub/data/ghcn/v2/grid/grid_1880_2008.dat.gz

    Then I went to the NCDC and read the README
    ftp://ftp.ncdc.noaa.gov/pub/data/ghcn/v2/grid/README_GRID_TEMP.txt

    It says:

    There are two options: grid_1880_YYYY.dat.gz, where YYYY is the current year and the values are calculated using the first difference method, and
    anom-grid-1880-current.dat.gz which uses the “anomaly” method.

    To me, it looks like you downloaded the wrong data set.

    You’re welcome.


    Steve: Excuse me. I provided the data set in two ways – the script is provided and shows the data set. Second I cite the data set and time here http://www.climateaudit.org/?p=4395#comment-312618

  60. Deep Climate
    Posted Nov 20, 2008 at 5:39 PM | Permalink

    I shouldn’t have to go through your script or read all the comments to figure out which dataset you downloaded; it should have been in the original post and it wasn’t. But that’s a minor point, as it wasn’t that hard to find it.

    The more salient point is it that it appears to be the *wrong* file.

    Steve: Compared with standards in the industry, surely you must concede that this is a model of transparency – and this is a blog. In “peer reviewed” articles, one seldom knows what data was downloaded and what version. But as to your point, I had already examined the file ftp://ftp.ncdc.noaa.gov/pub/data/ghcn/anom/anom-grid2-1880-current.dat.gz and it only goes into 2007. If you have some other candidate url, I’m all ears but please show the data set rather than just railing about it. The problem with your theory is that the match is OK for so many gridcells.

    I’m in the process of looking at
    ftp://ftp.ncdc.noaa.gov/pub/data/ghcn/v2/grid/grid_mean_temp_1880_current.dat.gz – it only goes into 2007 as well

    • Earle Williams
      Posted Nov 20, 2008 at 5:46 PM | Permalink

      Re: Deep Climate (#70),
      Could you point me to the URL for the “right” file, as suggested by your reading of the README?

      Thanks heaps!

      • Posted Nov 20, 2008 at 6:02 PM | Permalink

        Re: Earle Williams (#71), to expand, here is what the README file states:

        ftp://ftp.ncdc.noaa.gov/pub/data/ghcn/v2/grid/README_GRID_TEMP.txt

        There are two options: grid_1880_YYYY.dat.gz, where YYYY is the current year and
        the values are calculated using the first difference method, and
        anom-grid-1880-current.dat.gz which uses the “anomaly” method.

        Looking in the same directory as the README file, I cannot see any file with anom in the name:

        I would also appreciate it very much if Deep Climate could point out the correct anom file because, right now, I am drowning in Deep Thought

        — Sinan

  61. Steve McIntyre
    Posted Nov 20, 2008 at 5:51 PM | Permalink

    I shouldn’t have to go through your script or read all the comments to figure out which dataset you downloaded;

    Do you suppose that Deep Climate writes to Michael Mann or Ben Santer making similar complaints?

  62. Deep Climate
    Posted Nov 20, 2008 at 6:01 PM | Permalink

    Steve,

    The NCDC README explicitly says that dataset is based on “first difference” not anomaly, so it can’t be the correct dataset. Maybe you can generate the 2008 anomalies from the 2007 ones, by applying the 2008 first difference, or perhaps the 2008 anomalies are in another file.

    I don’t think I’m “railing” about anything, but merely pointing out that you appear to be working with a different dataset than you thought. I’d suggest you focus on solving your problem instead of reacting inappropriately.

    • Earle Williams
      Posted Nov 20, 2008 at 6:05 PM | Permalink

      Re: Deep Climate (#73),

      Just a link to the other dataset that you downloaded, that’s all I ask!

  63. Earle Williams
    Posted Nov 20, 2008 at 6:03 PM | Permalink

    You would think he would at least verify the existance of the presumed alternate file before accusing you of getting the wrong one. But lo, there is no anom-grid-1880-current.dat.gz in the v2 folder at the NOAA ftp site. Interesting little twist from someone who accuses you of not doing your homework.

  64. Posted Nov 20, 2008 at 6:09 PM | Permalink

    Wait a second: The directory list keeps changing:

    • Earle Williams
      Posted Nov 20, 2008 at 6:15 PM | Permalink

      Re: Sinan Unur (#77),
      I notice that the README.TXT which Deep Climate upon bases his scolding is dated Oct 28, 2004. When did GHCN v2 get released?

      • Posted Nov 20, 2008 at 6:19 PM | Permalink

        Re: Earle Williams (#78), this one gets more interesting. It looks like the file grid_1880_2008.dat.gz is no longer on the FTP server.

        Hmmmmmmm … Can’t wait to see what is next 🙂

        — Sinan

  65. Deep Climate
    Posted Nov 20, 2008 at 6:30 PM | Permalink

    #71 through 75
    It’s there all right. I’m looking at it (and I think Steve did too). The name of the archive is slightly different from the file. Probably they forgot to change the name of the archive the last time the file was renamed for clarity.

    i.e.,
    ftp://ftp.ncdc.noaa.gov/pub/data/ghcn/v2/grid/grid_mean_temp_1880_current.dat.gz
    contains
    anom-grid-1880-current.dat

    It goes up to March 2007 only. So one obvious solution is this:
    Take the October 2006 anomalies, add the October 2007 and 2008 differences to each grid cell, and see if that gets you the correct result. Or ask NCDC nicely for a file of 2008 anomalies.

  66. Earle Williams
    Posted Nov 20, 2008 at 6:46 PM | Permalink

    Don’t look now, but that file is gone too.

  67. Deep Climate
    Posted Nov 20, 2008 at 6:49 PM | Permalink

    Probably getting too many hits right now, but it’s still there. Be patient and don’t all download at once!

  68. Earle Williams
    Posted Nov 20, 2008 at 6:54 PM | Permalink

    heh, now it’s back. Same time stamp too.

  69. Earle Williams
    Posted Nov 20, 2008 at 7:01 PM | Permalink

    I don’t know if we just happen to be viewing the NOAA web site at the time when someone is investigating the issue Steve M raised with Russel Vose, but on the off chance that someone from NOAA is reading this site at this moment, please consider putting a revised README.TXT in the ftp directory to explain that things are temporarily under repair. Thanks!

  70. Deep Climate
    Posted Nov 20, 2008 at 7:03 PM | Permalink

    #83
    Um, the point is this whole post and discussion was based on the wrong data as far as I can see.

    It’s possible that the data matching the NCDC October temp anomaly graphic is not archived publicly because it’s still preliminary or it is archived somewhere else, or who knows what. Like I said, if this is a big deal for you, calculate it yourself, or ask the NOAA nicely for it. Or not.

    Not sure what’s happening with the anomaly file – sometimes I see it, sometimes not. But Steve has it anyway, so he could post it for you all.

    • Earle Williams
      Posted Nov 20, 2008 at 7:09 PM | Permalink

      Re: Deep Climate (#91),

      I haven’t plugged any of this into R so as far as I’m concerned it is pure speculation on your part that Steve M is using the wrong data. If you are willing to do the calculation you posit in #81 I would think you have something other than supposition on your side. Given the current state of NOAA data at their FTP site, I’m inclined to think Steve McIntyre is on the right track, to the extent that nowhere on the NOAA site is the “right” dataset available.

  71. Earle Williams
    Posted Nov 20, 2008 at 7:05 PM | Permalink

    Wow, over the course of two minutes of refreshing that directory it disappeared, reappeared, and disappeared again. Could be cache or proxy issues on my end I suppose, or it could be at NOAA end.

  72. Posted Nov 20, 2008 at 7:12 PM | Permalink

    Interesting sequence of events. Pseudonymous poster claims Steve used the wrong data file. When asked to point out where the *right* data file is, it can’t (ignoring hand waving about some old files). Then, files start disappearing from the FTP server. Good times, good times.

    Steve, could you take a look at this joker’s IP address and run a whois query on it, so our over imaginative minds don’t start conjuring up conspiracy theories? No need to post the IP address itself lest it be construed as a violation of privacy. Just the results of whois on that address.

    If you don’t have the utility on your computer, you can do it at http://whois.domaintools.com/

    — Sinan

    — Sinan

  73. STAFFAN LINDSTROEM
    Posted Nov 20, 2008 at 7:43 PM | Permalink

    #66 tty, that linking may be violating SMHI linking rules, or am I
    just misinterpreting…???[SILLY rules if you ask me…]

  74. Deep Climate
    Posted Nov 20, 2008 at 7:48 PM | Permalink

    #93, Earle Williams:
    “… it is pure speculation on your part that Steve M is using the wrong data.”

    No. The datasets are clearly described in the README.

    • Earle Williams
      Posted Nov 20, 2008 at 8:44 PM | Permalink

      Re: Deep Climate (#95),

      No. The datasets are clearly described in the README.

      The two datasets are hardly clearly described. If they were it would be clearly obvious whether a user of these data is supposed to subtract the current dataset from the anomaly dataset as you speculate, to derive the correct anomaly. Nothing in the README text files for either suggest that is the case.

      The description in the README file suggests that the grid_1880_YYYY.dat.gz is a standalone product, containing temperature anomalies for each grid cell for each month since 1880. If I could actually see the data files at NCDC right now I could download the grid_1880_YYYY.dat.gz and compare it to the anom-grid2-1880-current.dat.gz and see. But I can’t, because NOAA has removed/hidden that file from their web site. The fact that the data are not currently available from NOAA lends credence to the argument that something is indeed wrong with the data, as Steve McIntyre hypothesized. Your tenuous argument that in fact anom-grid2-1880-current.dat is the proper dataset to use is preposterous, as you state yourself it ends in 2007. Your argument is feeble and based on speculation. Without a comparison between grid and anom it remains speculative.

  75. kuhnkat
    Posted Nov 20, 2008 at 8:30 PM | Permalink

    Deep Climate,

    So, produce the right dataset as described in the README!!!!

  76. Deep Climate
    Posted Nov 20, 2008 at 9:56 PM | Permalink

    Earle #98:

    The description in the README file suggests that the grid_1880_YYYY.dat.gz is a standalone product, containing temperature anomalies for each grid cell for each month since 1880.

    Please read the README carefully:

    grid_1880_YYYY.dat.gz, where YYYY is the current year and the values are calculated using the first difference method, and anom-grid-1880-current.dat.gz which uses the “anomaly” method.

    Obviously, “first difference” is not the same as “anomaly”.

    Definition of “first difference”:

    The first difference of a time series is the series of changes from one period to the next. If Y(t) denotes the value of the time series Y at period t, then the first difference of Y at period t is equal to Y(t)-Y(t-1).

  77. Steve McIntyre
    Posted Nov 20, 2008 at 10:31 PM | Permalink

    I went out for Thursday squash league a couple of hours ago and since 6:51 pm Eastern (5.51 blog time) the following two files have been deleted:
    ftp://ftp.ncdc.noaa.gov/pub/data/ghcn/v2/grid/grid_1880_2008.dat.gz
    ftp://ftp.ncdc.noaa.gov/pub/data/ghcn/v2/grid/grid_mean_temp_1880_current.dat.gz

    No change notice has been posted by NOAA. This is all too much like Michael Mann. Public agencies should not delete data sets under discussion without posting a change notice. This is absurd.

    As we speak, there are NO relevant files remaining in this directory, despite Deep Climate’s claims that all is well at NOAA.

  78. Deep Climate
    Posted Nov 20, 2008 at 10:41 PM | Permalink

    OK, I think I’ve found what appears to be the same “anom” file at another location. At least it’s the same size and name and for this one the archive name and file name match to boot. But of course it’s also only up to March, 2007.

    ftp://ftp.ncdc.noaa.gov/pub/data/ghcn/anom

    I just noticed #94:

    Steve, could you take a look at this joker’s IP address and run a whois query on it, so our over imaginative minds don’t start conjuring up conspiracy theories? No need to post the IP address itself lest it be construed as a violation of privacy. Just the results of whois on that address.

    Wow, now that’s scary … I’m speechless.

    Steve: Nope. That’s what Michael Mann does, not me. BTW I already reported on this file only going up to March 2007 in http://www.climateaudit.org/?p=4395#comment-312731 so why are you bringing this up again. As we speak, there is NO file that contains correct information and various files containing incorrect versions have been deleted while we’ve been talking about it. How hard is it for you just to agree with actual facts?

    • Posted Nov 21, 2008 at 5:19 AM | Permalink

      Re: Deep Climate (#101), no need for drama.

      It is natural to wonder if the person who claimed that Steve used the wrong data file and the person who deleted the previously publicly accessible files on the FTP server are the same people.

      It was only due to pure random chance that I and Earle Williams happened to look at the contents of that directory at the time that we did and it is even more pure random chance that I thought of archiving time stamped screenshots of the directory listings on Flickr.

      snip

      I am not going to pursue this, but I wanted to emphasize that there was nothing nefarious about what I asked.

      — Sinan

  79. Steve McIntyre
    Posted Nov 20, 2008 at 10:42 PM | Permalink

    In the last 30 seconds ftp://ftp.ncdc.noaa.gov/pub/data/ghcn/v2/grid/grid_1880_2008.dat has been deleted.

    Hello, Viet Nam. You can talk to us, NOAA.

  80. Posted Nov 20, 2008 at 10:44 PM | Permalink

    RE Deep Climate #99, why would NOAA bother to post a table of first differences of temperatures? The level is of interest, as are anomalies relative to some base period. If for some reason someone wanted the first differences, these could easily be computed from the levels. However, the levels cannot be backed out of the first differences.

  81. Steve McIntyre
    Posted Nov 20, 2008 at 11:05 PM | Permalink

    For what it’s worth, I don’t believe that any of the data sets in question are calculated using the “first difference” method. The data set that I reported on – now deleted – could be seen to be derived from anomaly versions of Haparanda and Karesuando. As also at another gridcell that I checked. This is climate science and one cannot assume that readme’s are accurate. The only thing that counts is what you can replicate with the data and the data sets that I downloaded were anomaly data sets – I’m 100% sure of that – plus some strange screw-up. They have nothing to do with first differences.

  82. Deep Climate
    Posted Nov 20, 2008 at 11:20 PM | Permalink

    #104:

    If you say so … but have you tried this file?

    ftp://ftp.ncdc.noaa.gov/pub/data/ghcn/v2/grid/grid_1880-2008_RVose.dat.gz

    And have you read this description?
    http://www.ncdc.noaa.gov/oa/climate/research/ghcn/ghcngrid.html#data

    Sometimes the front door works better.

    So let’s see … I showed that you were using the wrong dataset, and now I think I’ve pointed you to the right dataset.

    You’re welcome. Later …

  83. Steve McIntyre
    Posted Nov 20, 2008 at 11:27 PM | Permalink

    The RVose file is form last February and is irrelevant.

    I corresponded with Vose today and in that email he didn’t know what the problem was.

    Please stop congratulating yourself for nothing, DC.

  84. Kazinski
    Posted Nov 21, 2008 at 1:19 AM | Permalink

    As a longtime computer programmer, I have to say I’m not that surprised at the volatility were seeing at the NOAA site. It is symptomatic of Jr. programmers or techs that just hate to be caught out in an error. It is also symptomatic of an immature organization without the proper controls to manage a production library.

    I’m quite sympathetic, any veteran programmer worth his salt has screwed up more applications and data files than they can count, me included. The difference is that after a while they realize why the controls and the QC process are in place and work with what is there when it is sufficient, or develop better controls when it is not.

  85. crosspatch
    Posted Nov 21, 2008 at 1:34 AM | Permalink

    “It is symptomatic of Jr. programmers or techs that just hate to be caught out in an error.”

    I would almost agree with that. I am responsible for providing similar products but in a different industry. My response to the discovery that the data might be “just plain wrong” would be an order to shut off access to it immediately. The difference is that I would also post a notice that the data had been pulled until it could be reviewed and verified. The credibility of the data is critical to the credibility of the agency in this case. If the product is bad, the producer is bad. Should something like this happen in anything I am connected with, my first responsibility would be to the consumer of the data. If I can not provide data that I am confident is valid, I would have no choice but to pull what is on the site until I can provide what I believe to be accurate data.

    But, as my first responsibility would be to the consumer, there would also be an explanation of what was going on. In this case the data have simply disappeared with no trace as if they never existed. While there is nothing unethical in and of itself in that action, it doesn’t give a transparent appearance when the product is questioned in such a public fashion. It also tends to stoke the appearance of an adversarial relationship with the consumer. There is a certain appearance of arrogance when data that is owned by the people is being deleted without comment to the people who own that data.

    If new data appear tomorrow with everything “fixed”, SteveM can be proud that he has done the world a great public service but at a personal level, if there is no explanation of what the problem was and how it was fixed, it is both frustrating for the researcher and sows seeds of mistrust. It leads one to wonder how many other such errors are there and if they will only be fixed if they are discovered in a public fashion. How they handle this can go a long way toward a community that actually wants to help them get it right, or a community that doesn’t trust a single number they publish. There is no reason not to be transparent. I think most here just want to get it right.

  86. StuartR
    Posted Nov 21, 2008 at 3:02 AM | Permalink

    As a layman observer, I find the data management in these agencies appear to be jawdroppingly fickle. For example in the games industry you’ll find that everyone (even artists ;)) are able to handle terrabytes of data with ease, and are able to track changes to Gigabytes of code and data and revert whenever necessary. I suggest that NOAA and GISS at least start investigating some freeware source management for code – CVS – it’s not the best but its free and at a professional standard, this site has some others:
    http://www.nongnu.org/cvs/

  87. Mike Bryant
    Posted Nov 21, 2008 at 4:29 AM | Permalink

    Yes, you have to wonder. Does NOAA want it right or do they simply want it warm, the truth be damned?

  88. Timo Hämeranta
    Posted Nov 21, 2008 at 4:46 AM | Permalink

    Re 45 Steve,

    I just sent you email with instruction to order Rovaniemi monthly mean temps 1951-Oct/2008 in digital form directly from the Finnish Meteorological Institute.

    Best

    Timo

  89. tty
    Posted Nov 21, 2008 at 7:18 AM | Permalink

    Re 95

    The link very likely violates what SMHI says about links on their website. However their claims wouldn’t hold in court. Swedish copyright law has a requirement about “verkshöjd”, i e that a work has to have a certain degree of originality, and couldn’t be done “by anybody” for it to be protected. They would have a hard time proving that a list of weather-station numbers is a work of art….

  90. krghou
    Posted Nov 21, 2008 at 9:32 AM | Permalink

    What’s the chance this blog belongs to Mr. Deep Climate?

    http://deepclimate.wordpress.com/

  91. Aaron Wells
    Posted Nov 21, 2008 at 9:39 AM | Permalink

    What’s the chance this blog belongs to Mr. Deep Climate?

    http://deepclimate.wordpress.com/

    From that website:

    I have identified myself at all times in dealing with all correspondents, who include many of the principals involved in the various topics presented here.

    If this is the same deepclimate, he/she claims to be totally open about performing some type of watch-dog analysis of the “skeptic” community in Canada. Yet I don’t remember him/her revealing this anywhere in his/her CA posts.

  92. Posted Nov 21, 2008 at 9:39 AM | Permalink

    Thank you, Sinan, for your equanimity and quiet fierceness. Steve deserves to have people with plenty of both on his epic journey. They said the age of dragons and dragon-slayers was over…

    Still around here someplace, Gollum?

  93. Steve McIntyre
    Posted Nov 21, 2008 at 9:44 AM | Permalink

    Look, I don’t care two figs about who Deep Climate is. Whether his points here are valid, that’s all that matters. Unfortunately so far his comments have so far consisted of rather over-confident slagging, nothing’s stood up and he expects to be thanked for saying incorrect things.

  94. Posted Nov 21, 2008 at 11:07 AM | Permalink

    Deep Climate as Obi Wan:

    “These aren’t the droids you’re looking for.”

  95. Sam Urbinto
    Posted Nov 21, 2008 at 11:44 AM | Permalink

    Steve knows how to sift through data and find what’s clearly wrong or probably right. As he has shown over and over and over again.

    Wait a second. There’s a climate science skeptic movement in Canada? Really! Do they serve cake and their meetings?

  96. Deep Climate
    Posted Nov 21, 2008 at 3:53 PM | Permalink

    See
    http://www.climateaudit.org/?p=4414#comment-312891 ff for the latest.

  97. Posted Nov 21, 2008 at 6:56 PM | Permalink

    Anomalouser yet, why are there many more “land”observations in the combined “land/sea” NOAA data seta than there are in the “land” data set depicted in the maps on http://www.ncdc.noaa.gov/oa/climate/research/2008/oct/global.html??

  98. Mike Bryant
    Posted Nov 21, 2008 at 8:59 PM | Permalink

    More like ANOMALOUSY…
    The whole thing is lousy… lousy agencies, lousy data, lousy individuals.

  99. Dan White
    Posted Nov 21, 2008 at 10:21 PM | Permalink

    “Deep Climate” scrambled up also reads “Ample Deceit.” Coincidence? … hmmm… OK it’s just a joke!

  100. Mike Bryant
    Posted Nov 21, 2008 at 10:38 PM | Permalink

    And “Steve McIntyre” scrambled up reads “Evict ’em Sentry”.

  101. Bill McNabb
    Posted Nov 22, 2008 at 7:32 AM | Permalink

    The November issue of the Communications of the ACM ( a leading computer science journal) is a tribute to Jim Gray. Gray was a pioneering database researcher. He had won the Turing Award which is the comouter sciecne equivalent of the Nobel Prize or Fields Medal. He disappeared in 2007 while sailing his boat in the Pacific off San Francisco.

    Gray would be of interest to this thread because he was building distributed databases on the Internet to contain the world’s scientific data. This description is from his home page at Microsoft Research

    Jim also is working with the astronomy community to build the world-wide telescope and has been active in building online databases like http://terraService.Net and http://skyserver.sdss.org. When the entire world’s astronomy data is on the Internet and is accessible as a single distributed database, the Internet will be the world’s best telescope. This is part of the larger agenda of getting all information online and easily accessible (digital libraries, digital government, online science …). More generally, he is working with the science community (Oceanography, Hydrology, environmental monitoring, ..) to build the world-wide digital library that integrates all the world’s scientific literature and the data in one easily-accessible collection.

    http://research.microsoft.com/~Gray/JimGrayHomePageSummary.htm

    Skyserver (http://skyserver.sdss.org ) began with making the Sloan Digital Sky Survey online and has been expanded to be able to contain large amounts of astronomical data. It is intended to malk all astronomical data available on line to any researcher with simple SQL queries. It is described in an article in the tribute issue of the Communications of the ACM. The article is by someone who collaborated with him in developing Skyserver. Of particular interest to this thread is the statement in the article, that old versions of data are never discarded or lost. If data is modified or added to, older versions are retained so that existing research can be confirmed.

    Also of interest is the principle that Gray followed that data manipulation should always be close to the data. So data should be able to be manipulated directly within the database. So instead of extracting data to be manipulated in custom R programs, for example, these statistical and other functions should be capabilities of the database itself. For example, the MBH 98 algorithm would be a capability of a climate science database. If some researcher, or some interested member of the public, wanted to see how robust MBH 98 was to the addition of new proxies then this should be able to be accomplished by a query to the database.

    DeepClimate appears to think the issue in this thread is whether or not some particular data is available. Gray’s work shows that to think this would be to miss the point. Data should be accessible and never be lost. Data manipulation should be facilitated by the database and be available to anyone. For example, one should not require years of specialist effort to be able to utilize algorithms around which current research is based.

    • Bill McNabb
      Posted Nov 22, 2008 at 7:45 AM | Permalink

      Re: Bill McNabb (#126),

      Note that NASA and the NSF are among the sponsors of the Skyserver database. So these agencies are familiar with the requirements of effective scientific databases.

  102. crosspatch
    Posted Nov 22, 2008 at 1:21 PM | Permalink

    Well, GISS isn’t *really* part of NASA proper. It is a lab of Columbia University Earth Institute. GISS was originally created in order to model the atmosphere of other planets but these days are pretty much limited in scope to only one planet. I am not sure whose names are on their paychecks. The website says “Close to half of our personnel are members of Columbia’s Center for Climate Systems Research (CCSR) and we also work with researchers at Columbia’s Earth Institute and Lamont-Doherty Earth Observatory.”

    I did notice this on the GISS page: “Systems and programming support for GISS is provided by Sigma Space Partners, a joint venture of Sigma Space Corp. and SGT, Inc.” A look at the link to Sigma Space Corp. had this on the “Services” page:

    Increases in computer technology and capabilities have enabled much more thorough, insightful, and expansive collection, access to, and analysis of data. Advances in the reliability of hardware and software have allowed 24 x 7 operations to occur with minimal human oversight. Our highly skilled personnel provide computer systems support in the areas of:

    * Scientific Computing (Algorithm and Application development)
    * Data Visualization
    * Operational Computing (System Development and Operations)
    * Data Archiving
    * Configuration Management
    * Database Design and Maintenance

    So maybe they can get their contractor on the stick.

  103. crosspatch
    Posted Nov 22, 2008 at 1:30 PM | Permalink

    Oh, apparently another function of their contractor is:

    Verification and Validation of science data systems and products

    Maybe they really should get that contractor on the case or maybe that contractor is where they got that “0.5” employee hours per month or whatever it was. Sounds like they have the resources available to them if they want to use them.

  104. aurbo
    Posted Nov 22, 2008 at 7:42 PM | Permalink

    Re #53:

    NOAA having Berlin 4 degrees C warmer is
    just ridiculous…Normal temp around 10C which it WAS…[+-0.5 deg]

    The October anomaly for Berlin reported in the WW&CB based on NOAA/NCDC data was +0.6°C. The rounded Temp was 10°C. Obviously there is/was a screw-up somewhere.

    • Posted Nov 23, 2008 at 11:49 AM | Permalink

      Re: aurbo (#130), That’s good info, aurbo. It has been said by others, but bears repeating: Most of these mistakes have tended to promote warming hysteria. They say “coincidence is God’s way of staying anonymous.” Maybe someone else thinks he’s God?

  105. Steve McIntyre
    Posted Nov 24, 2008 at 10:30 AM | Permalink

    A new version of ftp://ftp.ncdc.noaa.gov/pub/data/ghcn/v2/grid/grid_1880_2008.dat.gz was posted on the NOAA website today without explanation or apology.

    PS. It appears to be the same version as the one that was deleted last week. Gridcell 329 is the same as before and is still problematic.

  106. Sune
    Posted Nov 27, 2008 at 9:28 AM | Permalink

    Re: UC (#40)
    You show 3 time-series of temperatures in finland that follows the trend in the US you say. I am unsure of what the point is as 3/7000 stations proves nothing in terms of statistical significance.

    Then you talk about CRU and not GISS which was the target of my original question so the data you refer to is irrelevant. The fact of the matter is that GISS (not CRU) still claim that US tends (and western europe) have positive and negative trends while the rest of the world nearly exclusively show positive trends only.

    What we should be doing is this: Split all station data into two data sets:
    1) High quality station data: US plus western-europe
    2) Low quality station data: The rest of the world

    For each data set compute two numbers:
    a) Number of stations reporting a positive trend = Np
    b) Number of stations reporting a negative trend = Nn

    Then compute:
    R1 = Np(1)/Np(2)
    R2 = Np(2)/Nn(2)

    I will bet that R1 < R2 i.e. that US+europe have mixed trends while the more unreliable stations are biased toward positive trends.

    Make a graph of ad-hoc station-network quality (x-axis) and ratio of pos./neg. trends (y). Then you would se that only the low-quality networks substantiate the hypothesis of a global warming trend while high-quality networks will show equally many ups and downs.

    • Posted Nov 27, 2008 at 2:26 PM | Permalink

      Re: Sune (#134),

      I am unsure of what the point is as 3/7000 stations proves nothing in terms of statistical significance.

      Nah, 60 stations is enough, so USA represents one face of the hexecontahedron, and Finland another. 2/60 with no trend, 1/60 with a trend (central Europe, quite reliably measured warming there, I think).

    • KevinUK
      Posted Nov 28, 2008 at 7:35 AM | Permalink

      Re: Sune (#134),

      “Make a graph of ad-hoc station-network quality (x-axis) and ratio of pos./neg. trends (y). Then you would se that only the low-quality networks substantiate the hypothesis of a global warming trend while high-quality networks will show equally many ups and downs.

      What a great idea. I look forward to you doing exactly what you’ve suggested and reporting one the results you have obtained back here on this thread. Based on your results I’ll then consider emigrating to whichever temeprate climate country shows the greatest warming trend as a means of mitigating the onset of my arthritis :-).

      KevinUK

  107. Sune
    Posted Nov 28, 2008 at 4:36 AM | Permalink

    Re: #135
    The idea of discussions is to arrive at conclusions. You posts lacks preciseness and cannot be answered. Please read my response as I read yours. First you show 3 stations, then you talk about 60. Where is the proof and what are you assumptions? Can you, or can you not, reject the hypothesis that; US and western europe trends are different compared to the rest of the world? I want to see a p-value of a proper statistical tests as I proposed in #134. 1934 is still the warmest in US record. I expect to see a statistical (and verifiable) test in you next response. If not then this is not a scientific, but an emotional, discussion nobody needs. You could of cause also just admit you do not want to make the test. Then I would have to do it my self when time is available. That is certainly fair.

    • Jean S
      Posted Nov 28, 2008 at 6:05 AM | Permalink

      Re: Sune (#136),
      Calm down. It seems to me that you have not understood what UC was saying. In my opinion, your responses are also rather rude and lack respect. You are the one making the claim, so the burden of proof is on you.

    • John M
      Posted Nov 28, 2008 at 7:15 AM | Permalink

      Re: Sune (#136),

      Regarding the 60 sites, UC was drawing on a long-running joke.

      He was referring to a comment Gavin Schmidt has made, which UC further expounded on here.

  108. Posted Nov 28, 2008 at 7:17 AM | Permalink

    Sune,
    UC showed 3 stations that had a similar record to the US – temperatures in the 1930s being similar to the present day. But there are many other stations in many other countries that show the same picture. Go to the GISS station map and click and look for rural stations with 100-year temperature records. You will find a similar picture in Greenland (Angmagsalik, Godthab Nuuk) Iceland (Akureyri, Reykjavik) and Ireland (Valentia), with 1930s temperatures being similar to todays or higher.
    If you have time to do a proper statistical survey then that would be interesting.

  109. Sune
    Posted Nov 28, 2008 at 8:47 AM | Permalink

    Re: Paul #139
    Thank you for taking time the to explain the subtleties of the UC posts. I am quite new to this branch of “science” (climate science is a bit too mixed with political interests and green emotions for my particular taste) and it is clear that external auditing, as Steve McIntyre does, is long overdue. I am writing my Ph.d. thesis in an unrelated subject and I will look forward to do some proper statistical analysis on the subject of why US and rest-of-the-world seems to be in such discrepancy trend-wise. US should be large enough to pick up a global warming trend if it is there. Right now I think GISS is picking up high/low network quality. It needs to be investigated at least.

2 Trackbacks

  1. By "Anomalous" in Finland « Climate Audit on Nov 19, 2008 at 9:22 PM

    […] Original post by unknown […]

  2. […] http://www.climateaudit.org/?p=4395   Commenters who threaten anyone while here because they are not smart enough to come up with a better answer will have some due diligence done on them. Foul mouthed lefty posters beware. […]