Climate Audit

TCO

Where did you earlier discuss Hansen. There is no linnk or citation.

You also lack a link and citation to the Hansen pulbication itself.

Reading this it is unclear how much of the trend, you think comes spueriously from this artifact. Is it .10 degrees or what? Why don’t you give that?

Willis Eschenbach

Another oddity: why is the variance of the post 2000 so much greater than the pre 2000 data?

w.

reid simpson

is it possible that a switch “from USHCN adjusted to USHCN raw” would cause an increase in variance?

bernie

Willis:
Is the grey line actually the variance and if so the variance of what? It does not appear symmetrical about the mean value.
Thanks

SteveSadlov

Anyone remember that movie … “Entrapment.”

Not saying anything here, but the main theme of that movie was how a series of miniscule adjustments would result in the mother of all thefts.

Willis Eschenbach

Bernie, you say:

Is the grey line actually the variance and if so the variance of what? It does not appear symmetrical about the mean value.

Steve can correct me if I’m wrong, but I think the grey line is the data, and the black line is some kind of filtered average. As you point out, it appears asymmetrical, but I think that is because the top part of the data is cut off in this chart (again I may be wrong).

w.

• Steve Garcia
  Posted Jun 30, 2009 at 12:38 PM | Permalink

  Re: Willis Eschenbach (#12),
  Willis and Bernie –
  I enlarged the image and get three impressions:
  1.) Looking at other years and the data traces vs the smoothed out curve, it appears that this asymmetry is common on this graph.
  2.) It does not appear that the data line (grey) extends up past the top of the image.
  3.) It does appear that the data line (grey) has more width above the line than below, so I speculate that the data line above the black line represents more data points above than below. The negative spikes are sharper, implying single points, whereas the positive spikes are more rounded.

  My overall impression is that there are simply more positive values than negative ones, implying the positive spikes are multiple data points; i.e., there is some kind of a dwell on the positive side.

  It is too indistinct to tell for sure, but that is my impression.

  The data, of course, are the data. I am just adding to the discussion (way after the earlier comments) about the image.

Steve Garrison

Steve, So when are you going to publish this?

TCO

Ok, I visited it, now, and don’t see what is interesting.

Kenneth Fritsch

The NASA GISS Surface Temperature Analysis (GISTEMP) provides a measure of the changing global surface temperature with monthly resolution for the period since 1880, when a reasonably global distribution of meteorological stations was established. Input data for the analysis, collected by many national meteorological services around the world, is the unadjusted data of the Global Historical Climatology Network (Peterson and Vose, 1997 and 1998) except that the USHCN station records up to 1999 were replaced by a version of USHCN data with further corrections after an adjustment computed by comparing the common 1990-1999 period of the two data sets. (We wish to thank Stephen McIntyre for bringing to our attention that such an adjustment is necessary to prevent creating an artificial jump in year 2000.) These data were augmented by SCAR data from Antarctic stations not present in GHCN. Documentation of our analysis is provided by Hansen et al. (1999), with several modifications described by Hansen et al. (2001). The GISS analysis is updated monthly

I am eagerly anticipating the more later from Steve M. I need a reality check — again.

TCO

Wow. Good job, Steve.

I have not been keeping up with all the ins and outs, but think maybe a more prominent note is needed, if the are actually changin the data set? If a previous corrected version was incorrect and has now been replaced by a modifed one? Need to let users of the previous one know that they used an incorrect one?

mccall

Congratulations on the quick e-catch — words worth a thousand pictures.

JerryB

Re #20,

Doug,

The problem that Steve related in this thread is limited to
GISS usage of USHCN station data.

TCO

Donde?

TCO

If this data was used by a lot of other people, the change in correction may need to be stated in a publication. In that case, it may if central enough, warrent having you as a co-author given that you “made a significant contribution”. If that causes too much acrimony, since you won’t sign onto other things in their paper, or just don’t play well in the sandbox, it might be right for you to publish on your own.

All of the above is speculative as I have no idea about the datasets or their usage. I’m not sure if it’s a minor thing like eliminating a miscellaneous bad JCPDF, or if it is a big deal (widely used data for other work).

TCO

I don’t think it’s the last stone. Why do you think I think like that? I can disaggregate issues.

JerryB

Steve,

Let me add my congratulations. My guess is that the copy
to Gavin is due to his having been pelted by RC regulars
about your findings.

Regarding the GISSTEMP update:

Before:

“Input data for the analysis, collected by many national meteorological
services around the world, is the unadjusted data of the Global
Historical Climatology Network (Peterson and Vose, 1997 and 1998) except
that the USHCN station records included were replaced by a later
corrected version.”

After:

“Input data for the analysis, collected by many national meteorological
services around the world, is the unadjusted data of the Global
Historical Climatology Network (Peterson and Vose, 1997 and 1998) except
that the USHCN station records up to 1999 were replaced by a version of
USHCN data with further corrections after an adjustment computed by
comparing the common 1990-1999 period of the two data sets. (We wish to
thank Stephen McIntyre for bringing to our attention that such an
adjustment is necessary to prevent creating an artificial jump in year
2000.)”

The phrase “such an adjustment” seems yet to be defined.

TAC

SteveM: Excellent!

Everyone benefits when errors are caught and corrected.

This reminds me that it’s time to pay another visit to the Tip Jar… 😎

steven mosher

SteveM.

I’m dense. Did Reudy say that the mistake was .15C for the US over the period of 2000-2006?

That kinda renders the silly LLN debates moot.

The trend for the US is like .8C century.

Gosh. a error that big in just 6 years of data.

So, Anthony has fought global warming and reduced the Global temp by .01 by taking pictures.

What would that reduction cost in carbon credits

Hmm can we sell carbon credits for taking pictures?
.

steven mosher

RE 30. I think the Goddard guys deserve sme credit for doing the right thing.

I posted a thank you to gavin and crew on RC.

Josh Halpern and others made a big deal ( rightly) about a mistke that Kristin Byrnes made on a graph.
Some here advised her to correct her error ( science is a history of being wrong). She did.

And some gloated. I would advise against this. Goddard made a mistake. It was pointed out.
They fixed it. Congratulations on all sides.

Next?

Lee

[snip- I scrubbed something not because it was an error but because it was off topic and bickering. I’ve been trying to improve the threads by doing this and most readers appreciate the intervention with bickering people.]

Mikel Marià±elarena

This is what the contiguous U.S. GISS temperature history looks like now:

Warmest year now is 1934, followed by 1998, 2006, 1921 and 1931:

http://data.giss.nasa.gov/gistemp/graphs/Fig.D.txt

Will NOAA issue a press release to correct the one where they announced 2006 as the warmest year on record for the US?

http://www.noaanews.noaa.gov/stories2007/s2772.htm

Congratulations Steve!

Steve McIntyre

#42. Jerry, can you give me some particulars of changes and updates that you’ve noticed?

They have already changed their US (Figure D) numbers online without any preservation of the old numbers. By sheer chance, I happened to have the old information sitting in my active R session; I hadn’t saved them or planned to save them(I have now) and will post them.

JerryB

Steve,

I looked at three of the stations that I checked a few days ago, and
all three have completely new pre 2000 numbers in the GISS “raw” files.

Station Name . Number
CHEYENNE WELLS 051564
ENOSBURG FALLS 432769
HOPEWELL ….. 444101

GHCN Number
42572401001 HOPEWELL
42572465001 CHEYENNE WELLS
42572612001 ENOSBURG FALLS

JerryB

I’ve since checked Port Angeles, Eads, Hammon, Lakin, Boulder, and Kanab, all
of which have USHCN adjustments, and all have new pre 2000 numbers at GISS.

My guess is that they will change, or have already changed, the pre 2000 numbers
for all USHCN stations with non trivial adjustments.

Perhaps this is a temporary measure until USHCN version 2 is final, but that’s
just a wild guess.

Steve McIntyre

Here’s something interesting. If you compare “old” Hopewell VA numbers (fortunately preserved due to my much criticized “scraping” of GISS data) to the “new” Hopewell VA numbers, the GISS “raw” data for say June 1934 or June 1935 has gone up by 0.7 deg C, while the GISS “adjusted” data has gone up by only 0.1 deg C. So in some cases, their “UHI” adjustment as applied offsets what was a programming error. Makes you wonder about the validity of the UHI adjustment.

BTW as Jerry previewed, their US data set is now a total mess. Everything’s been written over prior to 2000.

Steve McIntyre

I’ve requested information through an email (this will be my 3rd request) but FOI time is close.

J Edwards

#52 I think it has to do with normalization of the surface data with the satellite data. At least that would make sense to me.

Dan Hughes

There’s an interesting read from earlier this year over at Open Mind on this subject.

Note the number of things that we were assured could not occur but have in fact now occurred.

BTW, does anyone know the actual facts regarding the TOB correction and the temperature data outside the US? I get the impression that the TOB “correction” has an important effect and that it is not applied to non-US data? Has the TOB model been updated and re-validated as new data have become available?

All this talk about “corrections” has made an incorrect application of the word readily accepted SOP. We can now make corrections when the correct answer is not known.

JerryB

Steve,

Reviewing Reto Ruedy’s note to you, I would revise my interpretation
in light of his statement:

“When we did our monthly update this morning, an offset based on the last
10 years of overlap in the two data sets was applied and our on-line
documentation was changed correspondingly with an acknowledgment of your
contribution.”

I would say that the average adjusments of 1990 through 1999 would be
what gets back out (“offset”) of 1999 and preceding years. For most
stations, the adjustments will be the same for each of those years,
but for some stations the adjustments may have changed.

I would say that such an adjustment would be a temporary measure. Whatever
is its rationale would not survive much of a critique.

Sam Urbinto

#55 Dan, It’s not just the best estimates thread, there’s also a brewhaha of sorts in the surface stations thread.

Liling calls 20% of the 125 year trend in just 6 years as being “a glitch”, but rather than deal with that, dhogaza complains about Dr. McIntyre instead!!

That glitch pales compared to McIntyre’s outright dishonesty about other issues, and Hansen’s graceful acceptance of the correction to the data analysis contrasts greatly with McIntyre’s unwillingness to acknowledge his own (frequent) errors.

Dano, Guthrie, Bloom, Boris and others are up to the usual tricks also. Maybe I should stop by.

David Smith

I plotted the annual GISS temperature versus the satellite-derived lower troposphere temperature (RSS) for the US. The resulting plot is here . The GISS temperatures include the recent adjustment.

The periods where GISS surges ahead of the satellite record appear to be associated with times of El Ninos , perhaps involving changes in precipitation. The record is muddied by major volcanoes in the early 1980s and early 1990s and any correlation is weak, but it’s my best guess.

The year 2006 looks like an outlier. It was about neutral on ENSO ( a weak La Nina and a weak El Nino occurred) so why the apparent surge? It looks odd.

Prior to the recent GISS adjustments the 2000-2006 period stood out as an odd period. Now, with the adjustments, GISS in the 2000s looks much closer to the satellite record (except for 2006).

Steve McIntyre

#65. DAvid, that’s a nice plot. I mentioned that I thought that there was still some air in the GISS 2006 numbers and this is another indication. There’s a major difference in data provenance in 2006: GISS only uses USHCN data up to March 2006 in their US numbers; they have a population of non-USHCN sites : primarily airports (which dominate the current data in all the indices). There is more up-to-date USHCN data available (up to late 2006), but, for some reason, GISS has not included it. MY guess is that inclusion of the USHCN data will lower the final US number (based on this graphic.)

M. Simon

#34 Bob Meyer,

I think “Butch Cassidy…” is more apt: “who are those guys?”

#46 Steve McIntyre,

You may have induced the “climate scientists” to commit a forced error.

J Weimer

Is there an elephant in this room? I find this discussion of statistical methodology fascinating, but somewhat reminiscent of rearranging the deck chairs on the Titanic. Pardon me, but I’m just an innocent interloper in this discussion. I undoubtedly don’t appreciate the finer points of statistical significance, yet with a few minutes of investigation, I’ve learned some things that are absolutely stunning, but have gone totally unmentioned in the blog dialogue, or the popular media.

(1) When the GISS talks about a “Surface Temperature Anomoly” they are not speaking about the surface of the earth, but rather about the tiny bottom layer of the air. But the air moves up and down in the atmosphere, so how can you estimate the heat content of the atmosphere by only looking at the bottom 6 feet?

(2) “Anomoly” means relative to the average temperature from 1951 to 1980 (a period of relative stability). That means that the 0.6 C temperature rise has occured in the past 25 years — since 1980, not since 1880, which is how the popular press reports it. They talk about temperatures rising over the past century and a half. What this data shows is far more drastic than anything they report.

(3) As startling as that revelation was to me, I saw something even more stunning when I looked at the GISS 2005 Summation. The global temperature rise is not evenly spread over the planet. It is highly concentrated in the North polar region, where it is actually 5 times greater than the average global rise — 2.5 to 3 C in the last 25 years! That observation should be hugely distressing but it is never the focus of the popular press coverage. They are much more concerned with the conflict and controversy over whether 1934 was a few tenths of a degree warmer than 2005, not whether the polar region has warmed 5.4 degrees F in the past 25 years.

I say the warming of the Arctic should be hugely distressing because that is the most dangerous region on the planet to experience a temperature rise. It is the one region where a temperature rise will create extreme and irreversible positive feedback. Wherever the North polar ice cover is removed to expose the ocean and the tundra, the sun’s energy will be absorbed at 10 times the rate, and the thawing tundra will release methane with 20 times the GHG effect of CO2. Any temperature rise in the Arctic will be amplified, accelerate exponentially, and become irreversible. In the face of this looming planetary catastrophe, what does it matter if the U.S. lower 48 was a fraction of a degree hotter in 1934 or 2005?

(4) I suppose that temperature records are the only data we have that allow us to look back a century or more but it seems to me that total atmospheric heat energy is what we’re really after, not temperature. That would require us to also know the density and moisture content of the air for every temperature measurement. It would also require us to estimate the total air mass assumed to be associated with each discrete point and time where we have a temperature reading. Can someone please explain to me how that could possibly be calculated in a moving dynamic atmosphere, with a set of data points that must have changed in both quantity and quality over the past 125 years.

Perhaps we should be less concerned with identifying trends over past centuries, where we are bound to have all kinds of data problems. We could focus more of our concern on the last 25 years, where we have identified a real and dangerous problem, and where our data is likely to be much more complete, consistent, and reliable. However, to get any media attention there has to be controversy and conflict, so perhaps Stephen M could highlight how correcting Hansen’s Y2K error has reduced the 25-year warming of the Arctic from 5.4 degrees (F) to 5.35 degrees (just guessing)– if, indeed, the correction of post-2000 mainland U.S. temperatures has any effect at all on the Arctic measurements. (I suggest using Fahrenheit because the general American audience of the popular media can understand a 25-year warming of 5 degrees F much better than a 125-year warming of 0.6 degrees C, which is what we’ve been told so far.)

Kenneth Fritsch

The article linked here and coauthored by Pielke, Sr. expresses in its conclusion what I have been repeatedly attempting to say about the importance of quality control problems exposed here at CA and SurfaceStations.

Click to access R-318.pdf

CONCLUSIONS. As Davey and Pielke (2005) documented and Peterson (2006) acknowledges, 926 | JUNE 2007 several USHCN stations are poorly sited or have siting conditions that change over time. These deficiencies in the observations should be rectified at the source, that is, by correcting the location and then ensuring high-quality data that are locally and, in aggregate, regionally representative. Station micrometeorology produces complex effects on surface temperatures, however, and, as we show in this paper, attempting to correct the errors with existing adjustment methods artificially forces toward regional representativeness and cannot be expected to recover all of the trend information that would have been obtained locally from a well-sited station.

The comparison of the reanalysis with the unadjusted and adjusted station data indicates that the reanalysis can be used to detect the inhomogeneity of individual station observations resulting from nonclimatic biases. In general, the adjustments indeed correct a large portion of nonclimatic biases in these poorly sited stations as far as the difference between the NARR/NNR and station data is concerned. The NNR yields a relatively uniform and statistically significant trend in this region, which is statistically similar to two of the four station trends. However, we found that there are some inconsistencies in the trends of the adjusted data. Among the four stations that have been subjected to adjustments, only the adjusted trend at Lamar is consistent with the NNR trend (being statistically similar). The other three adjustments either make the consistent trend (Cheyenne Wells) statistically inconsistent, produce a statistically significant larger trend than for the surrounding stations (Las Animas), or cause little change in the trend (Eads). This leads us to conclude that, whereas the adjustments do improve the consistency among the nearby station data and reduce the differences with respect to the reanalysis at the monthly and yearly scales, the trends of the adjusted data are often inconsistent among closely located stations.
Peterson’s approach and conclusions, therefore, provide a false sense of confidence with these data for temperature change studies by seeming to indicate that the errors can be corrected. For instance, the dependence of the corrections on other information (such as regional station moves, which in itself has been found on occasion to be inaccurate) can be considered an indication of the uncertainty and limitations of the “corrective approach” that is being sought. As a requirement, the statistical uncertainty associated with the effect of the adjustments on the regional temperature record needs to be quantified and documented.
Temperature adjustments such as those resulting from change in instrumentation are, of course, necessary. However, the results shown in this paper demonstrate that the lack of correctly and consistently sited stations results in an inherent uncertainty in the datasets that should be addressed at the root, by documenting the micrometeorological deficiencies in the sites and adhering to sites that conform to standards such as the Global Climate Observing System (GCOS) Climate Monitoring Principles (online at http: / /gosic.org/GCOS/ GCOS_climate_monitoring_principles.htm). A continued mode of corrections using approaches where statistical uncertainties are not quantified is not a scientifically sound methodology and should be avoided, considering the importance of such surface station data to a broad variety of climate applications as well as climate variability and change studies.

From the USHCN itself, we see from the excerpt below the special problem that undocumented changes (unknown noncompliances) can present to any adjustment schemes. It is interesting, however, that USHCN and NOAA documents and scientists, like the poster here at CA, Lee, make numerous references to quality control as that process for (attempting) extractions of valid data from poorly obtained collections. They spell out what they see as an optimum site for collecting temperature measurements, but on further reading it is clear that a true proactive quality control process is not in place. Below also is the process used for detecting and adjusting for undocumented changes. I would dearly like to find a statistically versed person who could determine, if nothing more than qualitatively, the assumptions that are made about the compliance of stations in general in using the adjustment processes.

http://www.ncdc.noaa.gov/oa/climate/research/ushcn

The potential for undocumented discontinuities adds a layer of complexity to homogeneity testing. Tests for undocumented changepoints, for example, require different sets of test-statistic percentiles than those used in analogous tests for documented discontinuities (Lund and Reeves, 2002). For this reason, tests for undocumented changepoints are inherently less sensitive than their counterparts used when changes are documented. Tests for documented changes should, therefore, also be conducted where possible to maximize the power of detection for all artificial discontinuities. In addition, since undocumented changepoints can occur in all series, accurate attribution of any particular discontinuity between two climate series is more challenging (Menne and Williams, 2005).
The USHCN Version 2 homogenization algorithm addresses these and other issues according to the following steps. At present, only temperature series are evaluated for artificial changepoints.
1. First, a series of monthly temperature differences is formed between numerous pairs of station series in a region. The difference series are calculated between each target station series and a number (up to 40) of highly correlated series from nearby stations. In effect, a matrix of difference series is formed for a large fraction of all possible combinations of station series pairs in each localized region. The station pool for this pairwise comparison of series includes U.S. HCN stations as well as other U.S. Cooperative Observer Network stations.
2. Tests for undocumented changepoints are then applied to each paired difference series. A hierarchy of changepoint models is used to distinguish whether the changepoint appears to be a change in mean with no trend (Alexandersson and Moberg, 1997), a change in mean within a general trend (Wang, 2003), or a change in mean coincident with a change in trend (Lund and Reeves, 2002) . Since all difference series are comprised of values from two series, a changepoint date in any one difference series is temporarily attributed to both station series used to calculate the differences. The result is a matrix of potential changepoint dates for each station series.
3. The full matrix of changepoint dates is then “unconfounded” by identifying the series common to multiple paired-difference series that have the same changepoint date. Since each series is paired with a unique set of neighboring series, it is possible to determine whether more than one nearby series share the same changepoint date.
4. The magnitude of each relative changepoint is calculated using the most appropriate two-phase regression model (e.g., a jump in mean with no trend in the series, a jump in mean within a general linear trend, etc.). This magnitude is used to estimate the “window of uncertainty” for each changepoint date since the most probable date of an undocumented changepoint is subject to some sampling uncertainty, the magnitude of which is a function of the size of the changepoint. Any cluster of undocumented changepoint dates that falls within overlapping windows of uncertainty is conflated to a single changepoint date according to
1. a known change date as documented in the target station’s history archive (meaning the discontinuity does not appear to be undocumented), or
2. the most common undocumented changepoint date within the uncertainty window (meaning the discontinuity appears to be truly undocumented)
5. Finally, multiple pairwise estimates of relative step change magnitude are re-calculated at all documented and undocumented discontinuities attributed to the target series. The range of the pairwise estimates for each target step change is used to calculate confidence limits for the magnitude of the discontinuity. Adjustments are made to the target series using the estimates for each discontinuity.

Nasif Nahle

Translation to English from the posts 76 and 77:

“Esto gracias a Steven McIntyre quien encontró un bug en los datos de la NASA (en los cuales se basan todas las publicaciones serias) debido al error del Y2K.)”

Translation: “This was possible thanks to Steven McIntyre, who found a bug in NASA data (on which all serious publications are based) due to the Y2K error).”

Jim Edwards

Steve M and all readers:

I suppose the horse has left the barn and already entered the glue factory, but can we not call Hansen’s error a Y2K problem ?

Calling it a Y2K error, while funny, will lead the lay reader to assume that the error lies at Microsoft’s feet, rather than NASA’s.

Sam Urbinto

Or how about this:

http://earthobservatory.nasa.gov/Newsroom/NasaNews/2002/200201317366.html

Mike A.

This stuff is practically Greek to me, but thank you Steve for finding the problem. I am currently experiencing a heated argument with another poster on another website concerning Global warming and the validity of the IPCC report and its followers and their agendas, etc. Whilst I am a mere average citizen, he is a reporter for a newspaper in Canada and his ego knows no bounds. I wish some of you smart guys could come and put him to sleep. It would help. Thanks for the interesting comments.