White House advisers greeted the new Climate Change Science Program(CCSP) assessment report like Bush advisers would have greeted a favorable report on WMD. Jane Lubchenco, administrator of the National Oceanic and Atmospheric Administration said: “It’s not just a problem for the future. We’re beginning to see the impact on our daily lives.” On the left is a picture of senior White House science adviser John Holdren pointing to a graph showing WMD-like impact of climate change on the U.S. electrical grid system, describing the results as follows:
The electricity grid is also vulnerable to climate change effects, from temperature changes to severe weather events…. The number of [U.S. electrical grid disturbance] incidents caused by extreme weather has increased tenfold since 1992. The portion of all events that are caused by weather-related phenomena has more than tripled from about 20 percent in the early 1990s to about 65 percent in recent years. The weather-related events are more severe, with an average of about 180,000 customers affected per event compared to about 100,000 for non-weather-related events (and 50,000 excluding the massive blackout of August 2003).
To be precise for realclimate readers – the picture, of course, does not show John Holdren explaining electrical grid disturbance incidents, but Colin Powell explaining WMD to the UN. (Colin Powell did not actually include a picture of US electrical grid disturbance incidents in his UN speech about WMD; that is stitched into the graphic.) The quote in question comes not from Holdren himself, from the US Synthesis Report page 58, though Holdren described the report as containing the “most up-to-date scientific findings on the impacts of climate change.”
The graphic of electrical grid disturbance incidents to which “Powell” is pointing is shown below in its actual CCSP size. (The size in the final report is considerably larger than in draft reports, presumably reflecting a conscious decision by the authors to emphasize this image. In addition, 2008 data was added on during the editing process, increasing the Hockey-Stick-ness of the image.)
Figure 2. From CCSP, Global Climate Change Impacts in the United States, Final Report, p 58. Actual size.
None of the “blue ribbon” reviewers (IPCC’s Susan Solomon being one of them) saw anything wrong with this graphic. However, on June 17, within a day of publication, blogger Warren Meyer spotted the absurdity of electrical grid disturbance incidents as a supposd climate WMD:
First, does anyone here really think that we have seen a 20-fold increase in electrical grid outages over the last 15 years but no one noticed? Really? Second, let’s just look at some of the numbers. Is there anyone here who thinks that if we are seeing 10-20 major outages from thunderstorms and tornadoes (the yellow bar) in the last few years, we really saw ZERO by the same definition in 1992? And 1995? And 1996? Seriously? This implies there has been something like a 20-fold increase in outages from thunderstorms and tornadoes since the early 1990’s.
Anthony Watts picked up Meyer’s find later that day here.
If one pauses to look at the graph for even a few minutes, it makes no sense. Climate change in the US in 2008 was not so severe that it could have remotely caused a 10-fold increase in “climate-related” electrical grid disturbances incidents. Meyer followed up his identification of this absurd CCSP claim by contacting John Makens, the holder of the electrical grid disturbance incident data – see follow-up post here. Makens described in the increase in reported incidents as being due to changes from prior “haphazard” data compilation and primarily due to “better reporting”.
In 1997, the EIA (and Makins himself) took over the compilation of this data, which had previously been haphazard, and made a big push to get all utilities to report as required. They made a second change and push for reporting in 2001, and again in 2007/2008. He told me that most of this slope is due to better reporting, and not necessarily any underlying trend. In fact, he said there still is some under-reporting by smaller utilities he wants to improve so that the graph will likely go higher in the future.
The unavoidable conclusion is that the electrical grid disturbance incident hockey stick is not a climate WMD and should never have appeared in the CCSP report.
Warren neatly picked this spitball off the wall, but it raises another interesting question. How did this false claim of climate WMD get into the CCSP report in the first place? And as an answer to this question, I’m interested in something more technical than calling the lead authors of the CCSP report a bunch of names, however richly deserved those names may be.
I’m interested in procedures. These assessment reports typically make quite a fuss about relying on the “peer reviewed literature”. This raises the question: in what “peer reviewed” literature did the electrical hockey stick previously appear? Both the chase and the answer proved interesting.
Let’s start with the most lurid claim (from the caption to the Electrical Hockey Stick):
The number of [U.S. electrical grid disturbance] incidents caused by extreme weather has increased tenfold since 1992. The portion of all events that are caused by weather-related phenomena has more than tripled from about 20 percent in the early 1990s to about 65 percent in recent years. The weather-related events are more severe, with an average of about 180,000 customers affected per event compared to about 100,000 for non-weather-related events (and 50,000 excluding the massive blackout of August 2003) [201].
[201] is the executive summary to a predecessor CCSP report: 201. Wilbanks, T.J., et al., 2007: Executive summary. In: Effects of Climate Change on Energy Production and Use in the United States [Wilbanks, T.J., V. Bhatt, D.E. Bilello, S.R. Bull, J. Ekmann, W.C. Horak, Y.J. Huang, M.D. Levine, M.J. Sale, D.K. Schmalzer and M.J. Scott (eds.)]. Synthesis and Assessment Product 4.5. U.S. Climate Change Science Program, Washington, DC, pp. x-xi.
The final version of this assessment report (4.5) was released in Feb 2008 and is online here. There is nothing in the reference that supports or even mentions a “tenfold increase” in electrical grid disturbance incidents. The language in the Executive Summary referred to in the US Synthesis Report says mildly:
How might climate change affect energy production and supply in the United States? The research evidence about effects is not as strong as for energy consumption, but climate change could affect energy production and supply (a) if extreme weather events become more intense, (b) where regions dependent on water supplies for hydropower and/or thermal power plant cooling face reductions in water supplies, (c) where temperature increases decrease overall thermoelectric power generation efficiencies, and (d) where changed conditions affect facility siting decisions. Most effects are likely to be modest except for possible regional effects of extreme weather events and water shortages.
Nothing about a tripling of weather-related incidents.
Let’s try a related reference from the Synthesis Report:
The electricity grid is also vulnerable to climate change effects, from temperature changes to severe weather events. [191] The most familiar example is effects of severe weather events on power lines, such as from ice storms, thunderstorms, and hurricanes. In the summer heat wave of 2006, for example, electric power transformers failed in several areas (including St. Louis, Missouri, and Queens, New York) due to high temperatures, causing interruptions of electric power supply.
[191] is another chapter in the same CCSP report (online here). 191 Bull, S.R., D.E. Bilello, J. Ekmann, M.J. Sale, and D.K. Schmalzer, 2007: Effects of climate change on energy production and distribution in the United States. In: Effects of Climate Change on Energy Production and Use in the United States [Wilbanks, T.J., V. Bhatt, D.E. Bilello, S.R. Bull, J. Ekmann, W.C. Horak, Y.J. Huang, M.D. Levine, M.J. Sale, D.K. Schmalzer, and M.J. Scott (eds.)]. Synthesis and Assessment Product 4.5. U.S. Climate Change Science Program, Washington, DC, pp. 45-80.
Again, there is nothing in the chapter that remotely supports the claim that there has been a “tenfold increase” in incidents or that the proportion of weather-related incidents has tripled.
The reference for the data in the graphic is “EIA, 216.” This is:
216 U.S. Energy Information Administration, U.S. Department of Energy [data]; assembled by Evan Mills, Lawrence Berkeley National Laboratory. Data available at http://www.eia.doe.gov/cneaf/electricity/page/disturb_events.html
Warren Meyer had already followed the link to http://www.eia.doe.gov/cneaf/electricity/page/disturb_events.html and found John Makens. This website does contain listings of electrical disturbance incidents e.g. here. A browse through the incident reports showed a lot of outages caused, unsurprisingly by winter snow storms – which, I guess, are another global warming WMD. Although the CCSP report refers to the failure of electrical power transformers “due to high temperatures”, problems were mainly associated with storms and winds; the incident in Queens’ appeared to be related to demand exceeding supply. The incident report (see page 137 refers to “voltage reduction” following a “public appeal” and did not directly mention a transformer failure “due to high temperature”.
There’s another clue in the reference:
data assembled by Evan Mills, Lawrence Berkeley National Laboratory.
For Pielke Jr, this is a bit like saying “data assembled by Michael Mann”. Pielke Jr has already criticized CCSP use of a hurricane-damage Hockey Stick from Evan Mills – see here , covered by John Tierney here. Here is Evan Mills’ hurricane damage graphic as rendered by CCSP:
Readers should read Roger’s article before reading the rest of the article. He identifies the provenance of the CCSP figure as Mills, E., 2005: Insurance in a climate of change. Science, 309(5737), 1040-1044, which he had savagely criticized in 2005 here. He observed that Mills, E. (2005) was not peer-reviewed, but was merely a “commentary” and that the provenance of the its data was, in turn, not peer reviewed, but “grey literature” from a workshop. In effect, there is a sequence of so-to-speak academic data laundering, in which non-peer reviewed workshop data (which was heavily criticized elsewhere) was used in a non-peer reviewed commentary in a journal that also published peer reviewed article, which is ratcheted up into a key diagram in an Assessment Report – without ever having been published in peer reviewed journal article. He asked :
But more problematically, why is a report characterized by Science Advisor John Holdren as being the “most up-to-date, authoritative, and comprehensive” analysis relying on a secondary, non-peer source citing another non-peer reviewed source from 2000 to support a claim that a large amount of uncited and more recent peer reviewed literature says the opposite about?
In diagnosing the CCSP use of the flawed Evan Mills hurricane damage data, Pielke observed that Mills was a CCSP coauthor. Once again, a climate scientist participating in the author team of an assessment report was advocating and promoting his own work – the sort of problem that we encountered with Michael Mann and the TAR hockey stick.
Here we have a second Evan Mills incident in the Synthesis Report – this time involving electrical grid disturbance incidents. But this one is even worse than the one that Pielke rightly criticized. In the hurricane incident, at least there was something published in the “Peer Reviewed Literature” as a commentary, however flawed, that the graphic could be traced to. In this case, there seems to be nothing.
CSSP policies state:
The US Synthesis Report describes its sources as follows (see page 6)
The foundation of this report is a set of 21 Synthesis and Assessment Products (SAPs), which were designed to address key policy-relevant issues in climate science (see page 161)… In addition, other peer-reviewed scientific assessments were used, including those of the Intergovernmental Panel on Climate Change, the U.S. National Assessment of the Consequences of Climate Variability and Change, the Arctic Climate Impact Assessment, the National Research Council’s Transportation Research Board report on the Potential Impacts of Climate Change on U.S. Transportation, and a variety of regional climate impact assessments. These assessments were augmented with government statistics as necessary (such as population census and energy usage) as well as publicly available observations and peer-reviewed research published through the end of 2008.
These policies do not permit the CCSP Author Team to construct their own special-purpose Hockey Sticks for the US Synthesis Report without prior appearance in the “peer reviewed literature”. This slipped by the “blue ribbon” panel, but was picked up within a day by bloggers. We’ve criticized procedures under which authors of assessment reports “assess” their own work (e.g. Mann and IPCC TAR) and this is one more, particularly egregious example.
CA readers will also be interested in review comments on the First (Public) Draft – see John Christy’s comments on the First Draft (and I doubt that subsequent editing has modified his opinion):
Frankly, I am stunned at this document. I see here not a scientific synthesis of the myriad of issues and views regarding climate change, but a straightforward, unscientific attempt at advocacy. It really must go back to the drawing board before people like me will take it seriously. …
This document is so pervasively misrepresentative of the science that a legitimate review requires a broader-based explanation of its shortcomings…
By letting a collection of one-sided authors, who share common views on the topic, serve as their own writers and final reviewers (hardly a peer-review process), they have created the opportunity for the complete dismissal of the whole project!
When I was asked about the moral, if any, of the Mann hockey stick debacle, I regularly commented that Assessment reports should not rely on authors to assess their own work – that’s an invitation for problems. Unfortunately, that advice – given in a constructive and not adversarial way- has been ignored and the present CCSP report provides a particularly vivid example.
Suzanne Goldenberg 
Climate Audit 
The US Synthesis Report and the Search for Climate WMD
White House advisers greeted the new Climate Change Science Program(CCSP) assessment report like Bush advisers would have greeted a favorable report on WMD. Jane Lubchenco, administrator of the National Oceanic and Atmospheric Administration said: “It’s not just a problem for the future. We’re beginning to see the impact on our daily lives.” On the left is a picture of senior White House science adviser John Holdren pointing to a graph showing WMD-like impact of climate change on the U.S. electrical grid system, describing the results as follows:
To be precise for realclimate readers – the picture, of course, does not show John Holdren explaining electrical grid disturbance incidents, but Colin Powell explaining WMD to the UN. (Colin Powell did not actually include a picture of US electrical grid disturbance incidents in his UN speech about WMD; that is stitched into the graphic.) The quote in question comes not from Holdren himself, from the US Synthesis Report page 58, though Holdren described the report as containing the “most up-to-date scientific findings on the impacts of climate change.”
The graphic of electrical grid disturbance incidents to which “Powell” is pointing is shown below in its actual CCSP size. (The size in the final report is considerably larger than in draft reports, presumably reflecting a conscious decision by the authors to emphasize this image. In addition, 2008 data was added on during the editing process, increasing the Hockey-Stick-ness of the image.)
Figure 2. From CCSP, Global Climate Change Impacts in the United States, Final Report, p 58. Actual size.
None of the “blue ribbon” reviewers (IPCC’s Susan Solomon being one of them) saw anything wrong with this graphic. However, on June 17, within a day of publication, blogger Warren Meyer spotted the absurdity of electrical grid disturbance incidents as a supposd climate WMD:
Anthony Watts picked up Meyer’s find later that day here.
If one pauses to look at the graph for even a few minutes, it makes no sense. Climate change in the US in 2008 was not so severe that it could have remotely caused a 10-fold increase in “climate-related” electrical grid disturbances incidents. Meyer followed up his identification of this absurd CCSP claim by contacting John Makens, the holder of the electrical grid disturbance incident data – see follow-up post here. Makens described in the increase in reported incidents as being due to changes from prior “haphazard” data compilation and primarily due to “better reporting”.
The unavoidable conclusion is that the electrical grid disturbance incident hockey stick is not a climate WMD and should never have appeared in the CCSP report.
Warren neatly picked this spitball off the wall, but it raises another interesting question. How did this false claim of climate WMD get into the CCSP report in the first place? And as an answer to this question, I’m interested in something more technical than calling the lead authors of the CCSP report a bunch of names, however richly deserved those names may be.
I’m interested in procedures. These assessment reports typically make quite a fuss about relying on the “peer reviewed literature”. This raises the question: in what “peer reviewed” literature did the electrical hockey stick previously appear? Both the chase and the answer proved interesting.
Let’s start with the most lurid claim (from the caption to the Electrical Hockey Stick):
[201] is the executive summary to a predecessor CCSP report: 201. Wilbanks, T.J., et al., 2007: Executive summary. In: Effects of Climate Change on Energy Production and Use in the United States [Wilbanks, T.J., V. Bhatt, D.E. Bilello, S.R. Bull, J. Ekmann, W.C. Horak, Y.J. Huang, M.D. Levine, M.J. Sale, D.K. Schmalzer and M.J. Scott (eds.)]. Synthesis and Assessment Product 4.5. U.S. Climate Change Science Program, Washington, DC, pp. x-xi.
The final version of this assessment report (4.5) was released in Feb 2008 and is online here. There is nothing in the reference that supports or even mentions a “tenfold increase” in electrical grid disturbance incidents. The language in the Executive Summary referred to in the US Synthesis Report says mildly:
Nothing about a tripling of weather-related incidents.
Let’s try a related reference from the Synthesis Report:
[191] is another chapter in the same CCSP report (online here). 191 Bull, S.R., D.E. Bilello, J. Ekmann, M.J. Sale, and D.K. Schmalzer, 2007: Effects of climate change on energy production and distribution in the United States. In: Effects of Climate Change on Energy Production and Use in the United States [Wilbanks, T.J., V. Bhatt, D.E. Bilello, S.R. Bull, J. Ekmann, W.C. Horak, Y.J. Huang, M.D. Levine, M.J. Sale, D.K. Schmalzer, and M.J. Scott (eds.)]. Synthesis and Assessment Product 4.5. U.S. Climate Change Science Program, Washington, DC, pp. 45-80.
Again, there is nothing in the chapter that remotely supports the claim that there has been a “tenfold increase” in incidents or that the proportion of weather-related incidents has tripled.
The reference for the data in the graphic is “EIA, 216.” This is:
Warren Meyer had already followed the link to http://www.eia.doe.gov/cneaf/electricity/page/disturb_events.html and found John Makens. This website does contain listings of electrical disturbance incidents e.g. here. A browse through the incident reports showed a lot of outages caused, unsurprisingly by winter snow storms – which, I guess, are another global warming WMD. Although the CCSP report refers to the failure of electrical power transformers “due to high temperatures”, problems were mainly associated with storms and winds; the incident in Queens’ appeared to be related to demand exceeding supply. The incident report (see page 137 refers to “voltage reduction” following a “public appeal” and did not directly mention a transformer failure “due to high temperature”.
There’s another clue in the reference:
For Pielke Jr, this is a bit like saying “data assembled by Michael Mann”. Pielke Jr has already criticized CCSP use of a hurricane-damage Hockey Stick from Evan Mills – see here , covered by John Tierney here. Here is Evan Mills’ hurricane damage graphic as rendered by CCSP:
Readers should read Roger’s article before reading the rest of the article. He identifies the provenance of the CCSP figure as Mills, E., 2005: Insurance in a climate of change. Science, 309(5737), 1040-1044, which he had savagely criticized in 2005 here. He observed that Mills, E. (2005) was not peer-reviewed, but was merely a “commentary” and that the provenance of the its data was, in turn, not peer reviewed, but “grey literature” from a workshop. In effect, there is a sequence of so-to-speak academic data laundering, in which non-peer reviewed workshop data (which was heavily criticized elsewhere) was used in a non-peer reviewed commentary in a journal that also published peer reviewed article, which is ratcheted up into a key diagram in an Assessment Report – without ever having been published in peer reviewed journal article. He asked :
In diagnosing the CCSP use of the flawed Evan Mills hurricane damage data, Pielke observed that Mills was a CCSP coauthor. Once again, a climate scientist participating in the author team of an assessment report was advocating and promoting his own work – the sort of problem that we encountered with Michael Mann and the TAR hockey stick.
Here we have a second Evan Mills incident in the Synthesis Report – this time involving electrical grid disturbance incidents. But this one is even worse than the one that Pielke rightly criticized. In the hurricane incident, at least there was something published
in the “Peer Reviewed Literature” as a commentary, however flawed, that the graphic could be traced to. In this case, there seems to be nothing.CSSP policies state:
The US Synthesis Report describes its sources as follows (see page 6)
These policies do not permit the CCSP Author Team to construct their own special-purpose Hockey Sticks for the US Synthesis Report without prior appearance in the “peer reviewed literature”. This slipped by the “blue ribbon” panel, but was picked up within a day by bloggers. We’ve criticized procedures under which authors of assessment reports “assess” their own work (e.g. Mann and IPCC TAR) and this is one more, particularly egregious example.
CA readers will also be interested in review comments on the First (Public) Draft – see John Christy’s comments on the First Draft (and I doubt that subsequent editing has modified his opinion):
When I was asked about the moral, if any, of the Mann hockey stick debacle, I regularly commented that Assessment reports should not rely on authors to assess their own work – that’s an invitation for problems. Unfortunately, that advice – given in a constructive and not adversarial way- has been ignored and the present CCSP report provides a particularly vivid example.