As we’ve discussed before (and is well known), clouds are the greatest source of uncertainty in climate sensitivity. Low-level (“boundary layer”) tropical clouds have been shown to be the largest source of inter-model difference among GCMs. Clouds have been known to be problematic for GCMs since at least the Charney Report in 1979. Given the importance of the topic for GCMs, one would have thought that AR4 would have devoted at least a chapter to the single of issue of clouds, with perhaps one-third of that chapter devoted to the apparently thorny issue of boundary layer tropical clouds.
This is what an engineering study would do – identify the most critical areas of uncertainty and closely examine all the issues related to the critical uncertainty. Unfortunately, that’s not how IPCC does things. Instead, clouds are treated in one subsection of chapter 8 and boundary layer clouds in one paragraph.
Interestingly, the language in IPCC AR4 is (using the terminology of climate science) “remarkably similar” to Bony et al (J Clim 2006) url , with the differences as interesting as the similarities. It seems to me that each language change from Bony to IPCC had the effect of papering over or softening the appearance of problems or contradictions, rather than clearly drawing the issues to the attention of the public. (Note – Bony was a lead author of the chapter – another instance of IPCC authors reviewing their own work.)
AR4
Boundary-layer clouds have a strong impact on the net radiation budget (e.g., Harrison et al., 1990; Hartmann et al., 1992) and cover a large fraction of the global ocean (e.g., Norris, 1998a,b). Understanding how they may change in a perturbed climate is thus a vital part of the cloud feedback problem. The observed relationship between low-level cloud amount and a particular measure of lower tropospheric stability (Klein and Hartmann, 1993), which has been used in some simple climate models and in some GCMs’ parametrizations of boundary layer cloud amount (e.g., CCSM3, FGOALS), led to the suggestion that a global climate warming might be associated with an increased low-level cloud cover, which would produce a negative cloud feedback (e.g., Miller, 1997; Zhang, 2004). However, variants of the lower-tropospheric stability measure, which may predict boundary-layer cloud amount as well as the Klein and Hartmann (1993) measure, would not necessarily predict an increase in low-level clouds in a warmer climate (e.g., Williams et al., 2006). Moreover, observations indicate that in regions covered by low-level clouds, the cloud optical depth decreases and the SW CRF weakens as temperature rises (Tselioudis and Rossow, 1994; Greenwald et al., 1995; Bony et al., 1997; Del Genio and Wolf, 2000; Bony and Dufresne, 2005), but the different factors that may explain these observations are not well established. Therefore, understanding of the physical processes that control the response of boundary-layer clouds and their radiative properties to a change in climate remains very limited.
Bony et al 2006
Boundary layer clouds have a strongly negative CRF (Harrison et al. 1990; Hartmann et al. 1992) and cover a very large fraction of the area of the Tropics (e.g., Norris 1998b). Understanding how they may change in a perturbed climate therefore constitutes a vital part of the cloud feedback problem. Unfortunately, our understanding of the physical processes that control boundary layer clouds and their radiative properties is currently very limited.
It has been argued based on the Clausius–Clapeyron formula that in a warmer climate, water clouds of a given thickness would hold more water and have a higher albedo (Somerville and Remer 1984; Betts and Harshvardhan 1987). But the analysis of satellite observations show evidence of decreasing cloud optical depth and liquid water path with temperature in low latitude boundary layer clouds (Tselioudis and Rossow 1994; Greenwald et al. 1995; Bony et al. 1997). This may be due to the confounding effect of many physical processes, such as increases with temperature in precipitation efficiency or decreases with temperature in cloud physical extent (Tselioudis et al. 1998; Del Genio and Wolf 2000).
Klein and Hartmann (1993) showed an empirical correlation between mean boundary layer cloud cover and lower-tropospheric stability (defined in their study as the difference of 700-hPa and near-surface potential temperature). When imposed in simple two-box models of the tropical climate (Miller 1997; Clement and Seager 1999; Larson et al. 1999) or into some GCMs’ parameterizations of boundary layer cloud amount [e.g., in the National Center for Atmospheric Research (NCAR) Community Climate System Model verion 3 (CCSM3)], this empirical correlation leads to a substantial increase in low cloud cover in a warmer climate driven by the larger stratification of warmer moist adiabats across the Tropics, and produces a strong negative feedback. However variants of lower-tropospheric stability that may predict boundary layer cloud cover just as well as the Klein and Hartmann (1993) parameterization, would not necessarily predict an increase in boundary layer cloud in a warmer climate (e.g., Williams et al. 2006 – Clim Dyn; Wood and Bretherton 2006 – J Clim).
The boundary layer cloud amount is strongly related to the cloud types present, which depend on many synoptic-and planetary-scale factors (Klein 1997; Norris 1998a; Norris and Klein 2000). Factors such as changes in the vigor of shallow convection, possible precipitation processes, and changes in capping inversion height and cloud thickness can outweigh the effect of static stability. These factors depend on local physical processes but also on remote influences, such as the effect of changing deep convective activity on the free tropospheric humidity of subsidence regions (Miller 1997; Larson et al. 1999; Kelly and Randall 2001). Evidence from observations, large-eddy simulation models, or climate models for the role of these different factors in cloud feedbacks is currently very limited.
The similarities are self evident. Now let’s look at the differences.
Bony et al said that boundary layer clouds had “strongly negative CRF” (Cloud Radiative Forcing), which IPCC watered down to “strong impact”. I guess that the idea of “strongly negative” feedback was too salacious for the IPCC audience.
| Boundary layer clouds have a strongly negative CRF (Harrison et al. 1990; Hartmann et al. 1992) and cover a very large fraction of the area of the Tropics (e.g., Norris 1998b). | Boundary-layer clouds have a strong impact on the net radiation budget (e.g., Harrison et al., 1990; Hartmann et al., 1992) and cover a large fraction of the global ocean (e.g., Norris, 1998a,b). |
The next sentence was identical other than trivial wordsmithing. Bony et al 2006 had stated that the “empirical” Klein and Hartmann (1993) correlation “leads” to a substantial increase in low cloud cover, which resulted in a “strong negative” cloud feedback. Again IPCC watered this down: “leads to” became a “suggestion” that it “might be” associated with a “negative cloud feedback” – the term “strong” being dropped by IPCC.
| Klein and Hartmann (1993) showed an empirical correlation between mean boundary layer cloud cover and lower-tropospheric stability (defined in their study as the difference of 700-hPa and near-surface potential temperature). When imposed in simple two-box models of the tropical climate (Miller 1997; Clement and Seager 1999; Larson et al. 1999) or into some GCMs’ parameterizations of boundary layer cloud amount [e.g., in the National Center for Atmospheric Research (NCAR) Community Climate System Model verion 3 (CCSM3)], this empirical correlation leads to a substantial increase in low cloud cover in a warmer climate driven by the larger stratification of warmer moist adiabats across the Tropics, and produces a strong negative feedback. | The observed relationship between low-level cloud amount and a particular measure of lower tropospheric stability (Klein and Hartmann, 1993), which has been used in some simple climate models and in some GCMs’ parametrizations of boundary layer cloud amount (e.g., CCSM3, FGOALS), led to the suggestion that a global climate warming might be associated with an increased low-level cloud cover, which would produce a negative cloud feedback (e.g., Miller, 1997; Zhang, 2004). |
The sentence starting “variants of the lower-tropospheric stability measure…” is identical in both versions.
Bony et al raised an argument about increasing albedo in clouds (dating back to the 1980s), noting three articles opposing this argument. IPCC deleted the mention of the arguments in favor of a higher albedo,
| It has been argued based on the Clausius–Clapeyron formula that in a warmer climate, water clouds of a given thickness would hold more water and have a higher albedo (Somerville and Remer 1984; Betts and Harshvardhan 1987), while keeping the three references to the opposing articles.
But the analysis of satellite observations show evidence of decreasing cloud optical depth and liquid water path with temperature in low latitude boundary layer clouds (Tselioudis and Rossow 1994; Greenwald et al. 1995; Bony et al. 1997). |
–
Moreover, observations indicate that in regions covered by low-level clouds, the cloud optical depth decreases and the SW CRF weakens as temperature rises (Tselioudis and Rossow, 1994; Greenwald et al., 1995; Bony et al., 1997; Del Genio and Wolf, 2000; Bony and Dufresne, 2005), |
Bony et al concluded their paragraphs reporting a very limited understanding of the physical processes controlling boundary layer clouds, a sentence that was substantially repeated by IPCC who qualified the admission of lack of understanding by saying that the understanding was limited in respect to “a change in climate”.
| Unfortunately, our understanding of the physical processes that control boundary layer clouds and their radiative properties is currently very limited. | Therefore, understanding of the physical processes that control the response of boundary-layer clouds and their radiative properties to a change in climate remains very limited. |
A third party reader might also assume that the section on boundary layer clouds would have benefited from comments from stadiums of IPCC reviewers. In fact, the version as published is almost word for word identical to the version in the First Order Draft. A few comments from reviewers were peremptorily dismissed by the chapter authors.
However, unlike the Hockey Stick section, there were virtually no comments whatever on this section and these were dismissed fairly summarily.
Reviewer Richard Allan observed:
8-586 A 47:54 48:5 It should also be noted that the cooling effect of clouds is primarily felt at the surface during the daytime, while the greenhouse effect of cloud generally heats the atmosphere. [Richard Allan (Reviewer’s comment ID #: 3-83)]
IPCC Authors:
Rejected due to space restrictions (this addition would not be fundamental for the following discussion).
My two cents worth as an interested non-specialist reader: Allan’s comment here seems interesting – it was something that I wasn’t aware of.
Next Allan suggested a seemingly interesting and on point addition to the text.
8-589 A 48:30 48:46 A suggested addition to the discussion of cloud altitude feedbacks: “Cess et al. (2001) [The influence of the 1998 El Nino upon cloud radiative forcing over the Pacific warm pool. J. Climate, 14, 2129–2137] suggested a strong influence of ENSO on cloud altitude and hence the balance between longwave heating and shortwave cooling. It is likely that this is partly a regional effect relating to changes in the vertical motion fields (Allan et al. 2002 [Influence of Dynamics on the Changes in Tropical Cloud Radiative Forcing during the 1998 El Nino J. Climate, 15, 1979-1986]) that may also be linked with decadal fluctuations in cloud properties (Wielicki et al. 2002 [Evidence for large decadal variability in the tropical mean radiative energy budget. Science, 295, 841–844.]) and is unlikely to be related to cloud feedback.” [Richard Allan (Reviewer’s comment ID #: 3-84)]
Again this suggestion was refused by the Chapter Authors.
Rejected. We do not review all the cloud feedback studies published, but assess the main progress that has been done since the TAR in understanding climate change cloud feedbacks. Therefore we do not discuss processes that are unlikely to be involved in climate change cloud feedbacks (e.g. the dynamically-driven change in clouds associated with El-Nino).
This latter excuse raises another interesting aspect of the paragraph on boundary layer clouds. Given the importance of the topic, a third party would assume that AR4 would include many references to a wide variety of studies since AR4 examining every conceivable aspect of marine boundary layer clouds. They rebuff Allan’s suggestion on the basis that they are assessing “progress since the TAR”. However, no fewer than ten of 13 references are pre-TAR (five pre-SAR) – there are only three references to post-TAR literature. Whatever the reason for excluding the Allan comment, it wasn’t because they were already chock-a-block with post-TAR literature.
As noted above, given the importance of clouds in climate sensitivity, and of boundary layer clouds in particular, a third party reader would have expected a comprehensive discussion of all the issues and, in particular, what steps they recommended for the reduction of uncertainties in this area, both of which were conspicuously absent.
Climate Audit 
104 Comments
Nice post, Steve. When rejecting the comment by Allen, the IPCC explained they “assess the main progress.” Apparently, they do not consider the cited works to be “progress.”
Re: Ron Cram (#1),
Perhaps progress is not a measure of scientific advancement, but of some other metric. I will ask Dr. Gore, I’m sure he will know.
Last rejection could be translated as “do you want us to cite a study which allows to conclude that we know even less than the little ammount that we said we knew before? Hah!”
A perfect example of how politicised the document is, ignore huge uncertainty that could cancel out the dire warming we want to predict. Ignore natural internal variability that could explain the observed warming trend etc…
As an engineer and modeller this kind of thing simply shocks me. If you dont fully understand something why even try modelling it???? Its a huge waste of time and money!!!
If there is huge uncertainty (and it’s potentially the difference between no change and decent warming) it needs to be clearly conveyed to the user (policy makers, the public and probably Al Gore) and upfront at the start of the document in clear text.
It SHOULD say “we really dont know what will happen due to uncertainty in cloud feedbacks, so we carried out sensitivity analysis with the following feed back values in the GCM’s and we get: a) no warming b) beneficial mild warming c) Significent warming. Due to large uncertainty, far more research is needed before we can confirm which scenario is most likely.” “Empirical paleoclimatic data, observation and logic would suggest a value somewhere between a) and b) with c) less being less likely.”
That is normal scientific practice. Respect those that search for the truth, doubt those that claim to have found it!
Fascinating. I’m surprised that the cloud feedback issue received so little reviewer attention. If you ask me, that’s pretty good evidence that the process isn’t working.
Great post and great question, stumpy.
At the risk of being snipped, the more I learn about the IPCC, the more they appear to be a body created for the express purpose of -snip
For me, the most stunning comment is “rejected due to space restrictions”.
To think that a report on ‘the greatest problem facing the planet and mankind’ is restricted by number of words! Particularly when you point out that clouds don’t get any where near their fair share of coverage anyway. Of course, that’s not the really reason, just a shoddy justification for not liking the comment.
If anyone is trying to download Chapter 8 from IPCC, there seems to be a problem with the file. I’ve managed to get it from archive.org.
Interesting that Bony was one of the lead authors on the chapter. This would explain the textual similarities. I guess it’s probable that there was disagreement among authors, which would explain the (ahem) change in emphasis, although I recall that chapter authors are meant to set out and explain scientific disagreements rather than papering them over in a sort of “consensus”.
Re: Bishop Hill (#51),
The IPCC report: what the lead authors really think
Thanks for the link — this is, indeed, an interesting and rather hopeful document. Some IPCC authors and staff are trying for self-correction. All quotes are from IPCC lead authors, responding to a 2007 questionnaire.
Samples:
I have no idea how much traction these IPCC lead authors will actually get — note that “The full details of [this] questionnaire and the replies submitted … have since been restricted.” And comments are (apparently) closed at this report site.
Happy reading–
Pete Tillman
Steve, I’ve finally understood why you get so irritated with posters ignoring the house rule that this Blog’s main function is to examine material used within the IPCC reports. It hadn’t occurred to me that the IPCC not only selected the reports that met their agenda but doctored the meaning of the ones they did use to better fit it. In effect the chapter authors are using these reports to lend the gloss of due diligence. I could be surprised that the authors of those reports weren’t more bellicous in calling them on this, I would hate my work to be misused in this way and would make my feelings known, but I guess being cited in IPCC reports carries more kudos than professional ethics.
.
On the other hand all the stuff that posters do cover here is really, really fascinating and on topic, I guess I at least just missed the point that a mining expert would know a rich seam when he smelt one.
.
Shocked, shocked I am, “space restrictions”, that is disgusting, why didn’t they just tell the reiviewer to FOAD that would have been more honest.
Ian
An editorial point: the words “while keeping the three references to the opposing articles.” are misplaced, they should be immediately after “deleted the mention of the arguments in favor of a higher albedo,”
I think the take-home message is that the IPCC 4AR is actually less informative about cloud dynamics and GCM modelling than the scientific reports to which it refers (and refers to poorly).
The most objectionable aspect of the whole IPCC process is the arrogant and dismissive behaviour of the authors towards the reviewers. Its as if a PhD thesis defence were reversed and now the candidate has right of veto of what he or she will allow as criticism from the PhD committee.
Re: John A (#11),
John A, I agree with your point about the IPCC chapter authors. It was interesting to see this attitude in a non-Hockey Stick section.
This isn’t the first time we’ve heard recently of recent non-supportive research being seemingly excluded from the discussion. This case is more interesting, because it relates to unknowns rather than contrary theories. I guess from the layman’s point of view, it is tempting to ignore the issues which are not understood and it might be difficult to get people excited about the omission of uncertainty. Is this just a trivial detail, or another part of the puzzle?
“The differences are self evident. Now let’s look at the differences”
Do you mean “The similarities are self-evident.”?
These suggest that cloud forcing is small:
Click to access i1520-0442-7-4-559.pdf
Click to access Harrison%20et%20al%20JGR%2095%20D11%2018687-18703%201990.pdf
some other info:
http://icp.giss.nasa.gov/research/data/erbe/
http://cimss.ssec.wisc.edu/wxwise/homerbe.html
Click to access week13.pdf
Re: thefordprefect (#14),
Note that these are old references. In a post a little while ago, I reported on more recent work (“superparameterization”) that indicated a substantial negative feedback.
Bony et al 2006 distinguish areas of upward convection where LW and SW approximately offset with subsidence areas where they don’t. In a quick perusal, the point in the cites seems to apply to the upward convection areas – not to the subsidence areas (areas with marine boundary layer clouds) that are the areas identified as being at issue.
The modeling issue is a fairly subtle one – models seem to systemically produce clouds that are too thick and cover too little area, thus underestimating the negative feedback.
My comments on this topic are merely an impression (as compared to my views on proxy-based issues which I’ve examined in depth).
It sure would speed up my publication process if I could summarily dismiss reviewer comments! I have to respond even to the ones that are incoherent.
For anyone old enough to remember, I cite Judy Collins “I really don’t know clouds at all”
Was Bony a member of the group writing Chapter 8? Otherwise this verges on plagiarism.
Re: tty (#18), Yes, she was. A couple of paragraphs before the one cited here, IPCC cites Bony et al 2006 as follows:
I only discussed boundary layer clouds in this post because that seems to be the major point at issue.
Kudos Steve. Reveiwing changes and comparing to the current state of knowledge is one of the duties of many of the engineers and other professionals who read your blog. I am sure each of us could regale you with our horror stories of how doing this saved our jobs because somebody was trying to “glad hand” a problem and get us to buy something that would not work. Instead, I would like to point out that 8-586 A 47:54 48:5 comment of Richard Allen is an accepted paradigm of why earth’s average temperature is about 13-16C. It is also the start of the general circulation of the GCM’s. It makes me wonder about the handoff from this poorly written section to the justification of the heat transport model they use to justify such statements as “we expected the Antartic to slow less cooling”, and other items that have been questioned.
Contrary to popular belief, lightning does tend to strike the same spot. You may have found a data rich vein here.
It is interesting that reviewers for journals insist that you cite work contrary to your results/theory, as do thesis advisers, but IPCC doesn’t have to. Are we going “post-modern”? Alan Sokol, where are you?
Thanks for the post, I wasn’t aware that the wording had been softened this much. You would think the IPCC would recommend more study in this area rather than water it out of contention. I wonder what the discussions were like before comments were dismissed.
Does anyone know if the IPCC have a process for a committee discussion of comments or is it one individual making the call? I’ve read the general structure before on their web site but not the detail of how it’s carried out.
Re: Jeff Id (#22),
Jeff, the Review Editor is charged under IPCC regulations for ensuring that Authors respond to Review Comments, but REview Editors seem not to have discharged this responsibility, instead acting as cheerleaders and letting Authors make their own decisions.
I and others have tried to obtain documentation on how Review Editors exercised their responsibilities and been rebuffed. IPCC has refused to disclose any Review Editor Comments. Efforts to obtain Review Editor comments on chapter 6 (where Hadley Center Chief Scientist John Mitchell was Review Editor) have been unsuccessful. In response to FOI, Mitchell first said that he destroyed all his correspondence, then he said that he acted as IPCC Review Editor in a “personal” capacity; when asked whether he got paid for his trips, they changed their story again and said that it would interfere with relations with an international organization (IPCC) to comply with FOI (an exemption).
In terms of process, it seems to me that the IPCC review process, despite all its self-praise, does not, for example, meet EPA standards.
Re: Steve McIntyre (#24),
Thanks Steve, it’s only the central issue for warming so I suppose glossing questions about it over is fine. If warming is what they say I sure want to know it, and that includes understanding feed backs. I’ve got too many projects going to dig into anything new right now but perhaps water vapor is next.
If anyone’s interested, Ryan was able to do a reconstruction using regularized least squares which has good results in comparison to RegEM and TPCA.
Recently making some press was the failure of airframe stress tests at Boeing that was not predicted by the computer design model. In aerodynamics, all the parameters are well known and not subject to chaotic conditions. How could such a clear cut computer model fail?
And yet, we are asked to believe that GCM’s will accurately predict climate for 100 years when one of the key variables is “not well understood”. Imagine if Boeing relied on computer models only when the tensile strength of the materials used was “not well understood”. Would you want to make the first flight on that aircraft?
Steve: Please dial back generalized complaining about GCMs. It’s not that there aren’t reasons for complaining but they’ve been expressed on many occasions and editorially it makes for “piling on”. Try to stay specific to the incident.
Re: Arn Riewe (#23),
The case you refer to was actually concerned with structures, not aerodynamics. That said it is indeed true that we have a very good, if not quite perfect, understanding of aerodynamics, and could, in principle “test fly” an aircraft in a computer model. However to solve the aerodynamics completely for such a complex shape as an aircraft over the whole flight envelope is very far beyond the computational capacity of extant computers. Therefore a lot of simplifying assumptions must be made which lessen the fidelity of the models. Even so computer modelling has now replaced a great deal of wind-tunnel testing. However there are things in aerodynamics that are still very difficult to model, flutter for example.
Modelling of clouds and precipitation are two sides of the same coin. The IPCC TAR has only one small image showing the precipitation anomaly. When you look at simulation of precipitation as depth rather than anomaly you get a very different picture. See:
href=”http://www.climatedata.info/Precipitation/precipitation.html”
and click on global precipitation.
Two things are noticable:
1. The average of the models is very different from the average observed precipitation.
2. The variation in modelled precipitation is much less than in the observed precipitation.
“This is what an engineering study would do”
I have seen such comments frequently on this site – demands for ‘engineering-quality’ studies, suggestions that ‘engineering studies’ are somehow the scientific gold standard, etc etc. They are not, and this statement doesn’t really make sense. A summary of science for public consumption has an entirely different purpose to an engineering study, obviously. So why should it in any way resemble one? Why would it be better if it did?
Re: RW (#27),
Simple, engineering studies are generally more professional and have more validity than an academic paper. Engineers make things that either work or don’t work…and everybody knows it when you fail. Academics are kinda like politicians with very little responsibility for being wrong.
Re: Jaye Bass (#31),
And they’re happy to tell you about it too. 😉
Re: RW (#27), Public consumption? You must be kidding. This document is intended to inform government policy and thus could affect every person on the planet. When the stakes are that large I don’t think it’s unreasonable to ask for “Engineering Quality” work. This does not mean it would be perfect, but would adhere to procedures which were accepted as “best practice”. If you are going to demand that everybody change their lifestyles I think we deserve that.
Re: RW (#27),
RW. I don’t know what your profession (if any) is. But have you ever heard of the term “Due Diligence?”
An excellent, informative and eye-opening post. It would go a long way in the effort to inform decision-makers if it were also available in pdf format that would allow for easier and more widespread dissemination. As a related note, the requirements of US law with respect to the response to comments raised during the promulgation of regulations are more stringent than the IPCC’s approach and are reviewable in court, if raised before the regulatory agency. The critical aspect of that legal requirement, however, is the identification of the data and analysis and/or scientific method that has been ignored. Posts, such as this can provide that critical aspect.
Here is a picture showing negative feedback from cumulus clouds up north in Norway. I shot this picure because it was very evident that the clouds did not form over the cold fjord.
Missing clouds over cold fjord
A quick bit of housekeeping, Mr. McIntyre: In the sentence, “…a third party would assume that AR4 would include many references to a wide variety of studies since AR4…”, I believe you meant that second “AR4” to be “TAR”.
Steve:
It’s your blog, and your time, but I for one find audits like this one and the penetrating study you, David Stockwell & others did of “Rahmsmoothing” far more valuable than the nattering on monthly cloud, ice and temp reports.
Regardless, keep up the good work!
Cheers — Pete Tillman
Consulting Geologist, Arizona and New Mexico (USA)
What about this “correction”?
Are they trying to say that the issue is a “global” problem rather than a “localized” problem? Or does Norris, 1998a,b discuss a different problem than Norris, 1998b does?
Seems they’re trying to increase the area of the “large fraction”, and not tie it to just the Tropics.
RE tty #10,
Thanks — I had trouble figuring out the comparison before your post. Bony in effect said that on the one hand, more clouds mean more albedo, but on the other hand they mean more heat-trapping. IPCC axed the one hand and only kept the other.
“engineering studies are generally more professional and have more validity than an academic paper”
Define “generally”, “more professional” and “validity” in this context, please.
“I don’t think it’s unreasonable to ask for “Engineering Quality” work”
Define “engineering quality”.
If you want argue that something should be different, you need to specify clearly what you want it to be, and you also need to explain why the new would be demonstrably better than the old. Simply saying that a science summary document should be “engineering quality” is a bit like saying the pasta I just bought in the supermarket should be “fish quality”. It doesn’t make any sense.
Re: RW (#40),
By example, an engineering quality study for a safety critical piece of software would be a defensible collection of tests and analysis that would certify that the software would meet the requirements of the system. I think that if you were in the business of making complicated things work you would know what was being talked about when someone mentions “engineering quality”.
Re: RW (#40),
Let me attempt to help you out here.
generally – meaning “in most cases” or “I would expect in the vast majority of cases”. This is a hedge because no doubt the original author has not seen “all” engineering quality reports. In my experience, every engineering quality report that I have seen has been “more professional” and has a higher “validity” (which we will get to next) than the academic submissions I have read, but I would guess that there are exceptional cases on both sides
more professional – meaning “to a higher standard”. In this case, being designed to stand up to serious design review and audit challenge. Reviews tend to be more formal affairs because if there is major property damage or loss of life caused by design failure, not only is the original author likely to be out of a job and possibly facing legal action, the reviewers will be in the same boat. It is the reviewers as well as the authors responsibility to see that all aspects of the design are covered faithfully and completely. In academic publication, there are many cases few penalties for braking or bending the rules.
NAS clearly said “Bristlecones are a very poor data set and should not be used”. Dr Mann used them anyway. Do you see any sign that his career was adversely affected ? Our host has issued specific and narrowly targeted technical objections about several of these papers that were simply ignored. In “professional quality” engineering assessments there are major career penalties for being wrong. In academic journals, reviewers are often more concerned about the politics of protecting their funding sources than the validity of the results. In an engineering quality report, much of the seemingly standard sophism and handwaving is not going to fly because the original author is not going to be able to avoid the technical discussion (which seems to be the “de rigueur” of climate science).
validity – meaning having higher objective content and more closely modeling the real world.
Have you considered that when discussing a highly technical subject matter with people who actually understand that subject matter, it might be wise to “learn the lingo” ? I would believe that the meaning of the term in question is standard and fairly obvious to those of us in the field. I really have no issues with your ignorance as we were all in the same condition at one point or another in our career, but do you not think it is a bit over the top to latch on to something that you don’t understand and then insist that it doesn’t mean anything to someone actually versed in the art ?
Re: Artifex (#44),
In your patient response, this phrase is excellent –
Some non-scientific and non-engineering people (to name but 2 groups) can have a concept of absolutism that is alien to good science and engineering. We do not usually state of a solution that “This is proven beyond all doubt and will never change” – because science has a history of progressing through the unexpected.
A strong theme that comes through Steve’s example is the lack of higher objective content and the lack of more closely modeling the real world. Indeed, it’s worse than that because it is reasonable to infer that deliberate attempts were made to avoid higher objective content and to avoid more closely modelling the real world.
Thank you for providing such an excellent explanation of “validity”.
Re: RW (#40), I will not let you tie me into semantic knots, nor do I intend to educate you on Quality Engineering practices. If you don’t know what this is, then it is your responsibility to seek the proper education yourself if you wish to be informed. I will give you a hint though, the IPCC assessment reports are not Engineering Quality products. Just the fact that the lead authors were not independent of the material being reviewed and were not required to resolve all reviewers comments disqualifies them. There’s more of course, but that’s enough by itself.
Re: RW (#40),
OK, I’ll take a stab at it. “Engineering quality” means that all of the work (including the analysis) in the study has been done at a level which meets or exceeds a predetermined standards appropriate to the study in question.
So how are the standards to be determined? For a start, you might try looking at the International Organization for Standard. They have over 16000 different standards drawn up in a large variety of areas including some for some types of statistical analysis . I would suggest that a report which adheres to these standards might be termed “engineering quality”.
So what “standards” exist for climate science papers? If we are supposed to rearrange the whole world based on arm waving, as you term it “fish quality” work (by the way, I’d bet that the pasta has to meet standards as well), and released to the media with great fanfare and questionable exaggeration, do you not think it might be reasonable to know that the work has met a reasonable level of reliability based on the use of accepted methodology? From numerous cases, peer review by fellow believers just doesn’t cut it. That is why current “audits” of the work are absolutely necessary.
By the way, I got a chuckle out of your manual for the television set. There’s not a lot riding on it, is there? Would you think that submitting the manual to the safety testing people was sufficient to justify that the television would not explode and burn your house down, or would you require that perhaps some stronger evidence be gathered (using testing meeting acceptable “engineering standards”) to justify that it would be safe for you to turn the TV on?
It is not surprising that Bony, as lead author, would have used text from a previous paper. After all, writing these chapters is a lot of work and the authors don’t have much spare time.
What is characteristic of the IPCC reports is that the chapter authors (especially the lead authors) use the occasion to highlight their own work, and usually ignore competing work. In the chapter on the carbon cycle, some scientists in the field who were not chapter authors did complain as reviewers that their work was not cited, and that important and misunderstood areas of the field were not discussed (this was again due to “lack of space”). In the end, when one compares the primary literature with the IPCC chapters, one usually gets a very different picture: evolving field with many remaining unknowns on the one side, settled science on the other. That is due, as is pointed out here, to a lousy review process, where the review Editor does not do a proper job. I don’t know if the IPCC report counts as a proper publication or not. In scientific circles, it wouldn’t surprise me if it were taken with a grain of salt, much like a review written for a popular magazine.
Re Francois Ouellette #41,
While a low-profile government report might not count for much, I’d imagine that being in the acclaimed IPCC report is worth a lot for the lead authors. After all, it means they share in a Nobel Prize!
To be sure, it’s the same Nobel Peace Prize Mother Theresa and Jimmy Carter won for their good intentions rather than their scientific contributions, but people will tend to forget that.
BTW (snip if OT), do IPCC expert reviewers like Steve and Ross now share in the designation as Nobel Prize Laureates, or just the lead authors? How was IPCC’s share of the money divided?
Re: Hu McCulloch (#42),
I recently attended a lecture by an IPCC4 lead author, and 3,000 people shared in that Nobel Prize. I don’t where the cutoff was, but I doubt there were 3K lead authors.
Apparently all authors got a slice of the pie. Don’t see why reviewers wouldn’t too.
Ultimately, IPCC-style science will be settled by linguists and literature professors.
Great to see the watering down process exposed. Did the same thing happen to solar influences?
Next – have the reviewers been picked for the next IPCC? How do we keep an eye on the next process as it unfolds? Or is it possible?
This must be Bony M. Is it “Brown Girl in the ring”, “Rivers of Babylon” or “Rah, rah Rasputin”?
The Spoonerism is more fun than the Bowdlerism, which borders on the unlawful. My old Dad used to say, long before global warming and trendy Lycra cyclists, “A well-boiled icicle”. Somehow reminds me of Antarctic Eric. Both ways. There might be a nurse ready to “prick his boil”. (self-snip…)
re 44. don’t forget the application of professional standards. no engineer would make up a novel statistical method where standard ones applied. and if we thought we had a new statistical method we would bring in a professional statistician to vet the method. and if he “vetting” the method we would document it fully so that others could use it.
A propos of the removal of important science on the grounds on “lack of space”, IPCC lead authors are apparently of the opinion that the next report should be shorter. (Link)
The rest of the piece is well worth a read. I was particularly surprised at the section entitled “Serious inadequacies in climate change prediction that are of real concern”.
Who knew?
Artifex – your definitions might equally well apply to the manual for my television set. It’s very well written, and everything in it is objective and corresponds exactly to what I observe on the real-world television set. So why shouldn’t IPCC reports be “TV manual quality”?
What you and everyone else is doing is meaninglessly trying to apply an irrelevant standards. Engineering and science are, in fact, very different things. Your laughable last paragraph only drives home that point. I am a scientist, publishing 2-3 peer-reviewed journal papers each year. Engineering reports have no relevance to how I do my research or the way I write it up, any more than TV manuals or any other form of documentation do.
“when discussing a highly technical subject matter with people who actually understand that subject matter”
I see no evidence that you actually understand climate science. This is not an engineering forum as you seem to think. I find it somewhat ironic that despite your fabulous claims of how perfect and precise engineering documents are, you are unable to clearly specify what this “gold standard” actually is.
Re: RW (#54),
And there it is in a nutshell, folks.
Re: RW (#54),
Do you make the source code and data publicly available for every peer-reviewed journal paper you publish?
Re: RW (#54),
Are you arguing that since something as mundane as a TV manual can be “well written”, “objective” and “correspond exactly to the real-world”, that these attributes are neither necessary or desirable in a climate science study?
Re: RW (#54),
“I am a scientist, publishing 2-3 peer-reviewed journal papers each year.”
Let me guess RW, you are a climate scientists right? LOL. I also a scientist bu thankfully not of the climatic variety. I’m been involved in a great deal of engineering studies and published many engineering reports in the nuckear and UK railway industries. I been involved in expert witness testimony at a number of public enquiries and I can assure you having closely followed the debate on climate change and read a greta many of the ‘Dano linkie’ report son this web site and others that climate scientist’s reports fall from short of the ‘engineering quality’ reports I’ve read and reviewed in the past.
KevinUK
Re: RW (#53),
.
I agree entirely. They are very different things. To encumber scientists with all of the baggage that comes along with engineering would be counterproductive and stifling. The purpose of science is to discover and learn; the purpose of engineering is to apply knowledge to create something useful. They are very different disciplines. (I do think that peer review in climate science has taken on a bit of the good ol’ boys club mentality, and some of the things that get published are really quite a joke, but your point is apt.)
.
With that being said, the IPCC reports are not scientific reports. The IPCC report (and others) are supposedly decision-making tools for policymakers. People expect such reports to be put together with engineering rigor, not scientific rigor. That means that the report should mention things that are known and spend all of the rest of the words on discussing what is not known. Failure to belabor things that are not known is irresponsible. Failure to discuss how things that are not known could affect the recommended actions is negligent.
Re: Ryan O (#60),
I think that in using and judging the worth of the material presented by the IPCC on climate change it is important to recognize what it is that the IPCC actually is. You note that reports from the IPCC on this subject are supposedly decision-making tools for policy makers and I would take that “supposedly” all the way to what we can surmise their mission actually is as judged by how the material is presented.
I would surmise from my view of their methods that the mission is to show evidence for the case for immediate AGW mitigation. I have made the past observations that the evidence is being presented as though it were an adversarial court case in the US system of justice with only one side being presented. I think, that after some thought, I need to modify my view to an adversarial court case with both sides (kind of) being presented with the authors inputs and the comments which may or may not be counter evidence or arguments to the authors. The hooker in the process would appear to be that the authors are in effect the lawyers putting on their case and the judge in proceedings.
I see the major problem with the IPCC methods in these matters as being presented as something different than what they supposedly are. I would have no problem if the IPCC reports on climate change were presented as what I think they are: a body for presenting evidence to make a case for immediate AGW mitigation – and by implication telling the reader to look elsewhere for the countervailing evidence.
Re: Kenneth Fritsch (#70), I think you have hit upon an important point that goes all the way back to the creation of the IPCC and its supposed mission. The people tasked with the advice to states sections are in a bit of a bind. The nation states have basically asked them to examine the science and tell them “What to do.” It is almost humanly impossible for such a group to write a report that says: “We can’t agree and don’t know what the scientific answer is and therefore we can’t give you any advice except to wait and see and do nothing now.” Their client states would no doubt respond with: “What the hell kind of answer is that! You’re out of business.”
Whether the result of this tension represents a corruption of the intended process, I don’t know. It just seems inevitable to me that the advice sections, regardless of a particular bias-or the lack thereof, are always going to come down on one side or another. Otherwise, nobody gets paid. The dice are always loaded.
Perhaps the only possible solution is to do away with the advice section; have the IPCC produce an objective, thoroughly scientific report, with room for all sides to contribute, and then submit the whole thing to entirely independent mediation.
On another level, a serious and perhaps insurmountable difficulty is that the whole IPCC process purports to give up an individual nation state’s rights to control its own destiny in a crucially important area. It would be difficult to think of another area in which so much apparent prestige is granted to the UN. Fundamental disputes are inevitable.
Re: philH (#101),
Unfortunately, Steve doesn’t allow discussion along these lines, so you’ll have to find another venue to get into the political aspects of IPCC and AGW in general. (Not that there aren’t plenty of such sites out there).
Re: RW (#53),
I’ve been asked to define “engineering quality” on several occasions and have not attempted to do so, though I can point to many differences.
I’ve seen engineering reports on mining projects (and I’m sure that an engineering report for an automobile model or power plant would have a generally similar sort of look). They are big, expensive studies costing millions or tens of millions of dollars. Key issues may themselves include sub-studies.
They look totally different than anything produced in climate science.
I’m not saying that the authors of such reports are “better” than authors of articles in the Peer Reviewed Litchurchur, only that they are doing something different.
In an engineering report, engineer don’t try to do anything “original”, whereas “originality” is a criterion for publication in the Peer Reviewed Litchurchur.
IPCC has construed its terms of reference to be a review of literature since the prior study. It does not do any due diligence on that literature, relying entirely on journal peer review – which is a process with an entirely different function. As a result, findings are reported in IPCC reports that have never been subjected to any third party due diligence. There is no proper chain of custody of results through to public policy. This does not mean that the “main” IPCC points are wrong. However, these “main” points are then mixed in with a mush of perhaps non-germane material with no markers as to what is “main” and what isn’t.
An engineering report on climate change would, for example, include a section on the infrared physics of CO2 absorption – and how increased CO2 actually causes enhanced greenhouse effect and how it interacts with water vapor and clouds. There is no such section in AR4 (nor for that matter in any previous IPCC report.)
An engineering report, as others have noted, would not be written by someone with an intellectual axe to grind in a dispute. The axe grinder may well be right, but society has learned that, for decision-making, it’s far better to have a truly independent review by someone competent and independent.
There are many other points of difference.
Engineering is what happens after the science is settled. You wouldn’t build a power plant – even a green power plant – based on a Peerreviewedarticle in Natureorscience.
Climate modeling has much akin to sophisticated engineering models – aircraft or space stations are often mentioned. ARguably climate modeling is more difficult. But that’s an irrelevant beauty contest. For now, I merely point out the difference in forms of reporting. A space station model would be reported in a long technical report, not a few peerreviewedarticles in Natureorscience.
RW, it’s not useful to react in a defensive way when such points are made. TRy to understand them for what they are. In my opinion, much of the frustration that the public has with climate scientists (and climate scientists with the public) arises out of the failure of climate scientists as a community to understand forms of engineering reporting.
RW – how does “something that is auditable” sound?
RW,
As an engineer, I’ve also at times had trouble understanding what SM means by “engineering quality”. Some of the things I’ve seen engineers turn out were masterpieces while others were borderline garbage. And depending on the level of analysis, the quality and accuracy of it was all over the board. If it’s a purely conceptual study, then there’s often very little information included and errors can be outrageously high (assuming it’s physically possible in the first place). Then again, if you’re building an aircraft, the documentation is extensive!
I’ve come to the conclusion that what SM means by “engineering quality” is that each step is detailed as to what is happening and what is assumed and then the effect of those assumptions are investigated and some sort of accuracy is determined for those assumptions. The information is then clearly documented (with references) in order for others to see what has been done.
I also don’t really consider this an engineering or science forum… I see this as a statistics forum. SM would be the first to say that he’s no climate scientist, but that doesn’t invalidate any of his issues as the issues raised here are typically related to poor data manipulation practices.
I’ll throw in my two cents about engineering quality work because the whole concept arose from the hit you in the head with a brick obviousness of the mathematical problems in Paleo reconstructions.
Consider that the field is rife with math that cannot even under the best circumstances, extract the signal from the noise yet that is exactly what they purport to do. All it takes is a reasonable check of the math and over and over we find the signal is the math. These people as a group regularly verify each others work, yet they have to know by now the methods don’t really work. I like the idea of ‘engineering quality’ but how about shooting for a lower bar like reasonable verification of method.
Re: Jeff Id (#63),
Bravo, Jeff! They fail to akcnowledge that a demonstrable signal is necessary to begin with.
Example engineering report …
Rehabilitation and Repair of Reinforced Concrete Short Columns With External Steel Collars.
338 pages, 4.7MB pdf.
Steve: This is NOT what I had in mind though I’d prefer this sort of technical report to what we get. This is a university report not a project report by Bechtel. A Bechtel engineering study on a power plant would look totally different than this,
I’m an avid lurker and first time poster. The quality and transparency of Steve’s (and others) work presented on this site are outstanding. I’m a geologist and geophysicist and have focussed on numerical modeling of fluid flow and contaminant transport (groundwater, surface water, and air)for nearly 25 years. I’ve always been facinated by climate science and I’ve been following its development, primarily in AGU journals, throughout my career.
I don’t wish to bash GCM’s or their utility (they ARE highly useful) but as a modeler (no Right Said Fred puns) I’m increasingly shocked at the level of claims regarding validity and predictive ability without reference to inherent uncertainties.
RW:
In regards to the engineering vs. scientific study issue: I routinely provide scientific studies (modeling predictions, uncertainties, etc) to engineers for design work. There are different levels of work quality that are needed depending of the criticality and phase of a particular project (is human health or safety a consideration, the $ involved, who are the stakeholders, etc). Sometimes back of the envelope type calcs are sufficient. At other times, rigorous analysis with complete documentation and QA/QC is necessary. The “criticality” that climate change has attained due to both fiscal and politcal implications should require a very high standard of quality.
This quality should include:
Documentation
Softare validation, verification, and documentation
Data Sets – from raw to final stages
Transparent and detailed descriptions of all data processing
Transparent and detailed descriptions of all calculations
Review
All comments must be addressed
Comments should be reconsiled between all parties (not always possible)
3rd party reviews (complete non-involvement)
Hostile reviews should be welcome.
My 2 cents
Steve and others – keep up the good work
Yes. What we are witnessing now with cap and trade and EPA findings is social engineering based on incomplete and/or suspect science.
RW: You wrote:
“Artifex – your definitions might equally well apply to the manual for my television set. It’s very well written, and everything in it is objective and corresponds exactly to what I observe on the real-world television set. So why shouldn’t IPCC reports be “TV manual quality”?”
Actually device manuals are often littered with errors and they are not published according to any professional standards.
You continue:
“What you and everyone else is doing is meaninglessly trying to apply an irrelevant standards. Engineering and science are, in fact, very different things. Your laughable last paragraph only drives home that point. I am a scientist, publishing 2-3 peer-reviewed journal papers each year. Engineering reports have no relevance to how I do my research or the way I write it up, any more than TV manuals or any other form of documentation do.”
you exactly miss the point here. What we are saying is that your science writing ought to rise to the level of an engineering standard. We are not applying meaningless standards. We are saying that “engineering” standards would improve the quality of science writing. in particular the practices in climate science of inventing and applying novel statistical methods in the same paper. very shoddy.
In my experience, RW’s attitude is based on the insular nature of the academy and associated conceits regarding academic vs. industrial enterprise. When I got out of grad school and went into industry I was floored by the change in expectations and standards. I soon learned that my published papers (peer-reviewed!) were no where close to what would be required to justify a significant corporate spending decision, for example.
Academics think that once a paper has passed peer-review it automatically achieves the status of TRUTH. In reality, when a paper passes peer-review it has achieved status of “NO THREAT TO CURRENT FUNDING REGIME”.
RW #53:
Let me cite the following 2 paragraph from IPCC AR4:
“Based upon the revised (Edition 3_Rev1)ERBS record (Figure 3.23) outgoing LW radiation over the tropics appears to have increased by 0.7 W/m2 while reflected SW radiation decreased by roughly 2.1 W/m2 from the 1980s to 1990s (Table 3.5)”.
“Since most of the net tropical heating of 1.4 W/m2 is a decrease in reflected SW radiative flux, the change implies a similar increase in solar insolation at the surface that, if unbalanced by other changes in surface fluxes, would increase the amount of ocean heat storage.
In summary, although there is independent evidence for decadal changes in TOA radiative fluxes over the last two decades, the evidence is equivocal. Changes in the planetary and tropical TOA radiative fluxes are consistent with independent global ocean heat-storage data, and are expected to be dominated by changes in cloud radiative forcing. To the extent that they are real, they may simply reflect natural low-frequency variability of the climate system.”
For my humbled opinion this suggests that cloud cover could well be the main cause for a lot of warming in the 1980s to 1990s.
However cloud cover change is not mentioned as a cause in the attribution section of the IPCC AR4 Key findings. How is it possible that the IPCC ignores its own findings? I don’t think the report meets engineering or scientific standards.
Having been a scientist as well as an engineering manager, I don’t think the example above from the IPCC AR4 would meet my standards, neither as a scientist nor as an engineer.
Best regards
Guenter
I was fascinated to read this in the doctor’s office last week:
New Scientist 2 May 09
Dear Steve McIntyre:
Your post is excellent. I like it a lot. Thank you very much.
You do what the IPCC should do with a lot of datasets and data.
I support JoseBBs #65 comments wholeheartly.
An report with the impact of the IPCC AR4 should meet the highest standards possible.
I think it would be resonable to publish standards as a guidelines for scientific papers that have to be met in order to be considered within an IPCC report. Like calibtration of the instruments, measurement system analysis according to standard qualtity management handbooks, etc. Independent measurements, etc. They could just use what is available in industry.
Thanks for your work
Best Regards
Guenter
In regard to the question about the relationship between an engineering study and science, wouldn’t the documents prepared for major scientific projects such as the Large Hadron Collider, a Mars probe etc be germane. They must have to combine science and engineering to make policy proposal for government funding and action.
What would the proposal document look like for one of the major experiments in the LHC. Wouldn’t it have t review the science and make scientific proposals to prove its scientific merit. Wouldn’t it have to specify the experimental apparatus to the extent that budgets can be set and feasibility projects specified. Wouldn’t it have to be of such clarity that the spending of billions of dollars could be justified?
This to me is engineering quality. Are the IPCC documents of such quality?
================
More importantly, are they of adequate form? Do they review the scientific work in such a way as to assess it for further funding?
Has enough progress been made?
Is the current scientific strategy the correct one?
What new opportunities are there to be exploited?
Are the current personnel of sufficient capability or should new personnel be chosen?
Any senior engineering manager would be very familiar with information requirements such as this. Do the IPCC documents contain sections of this form?
Re: TAG (#75), One of the requirements is that it is readable for the intended audience. As someone who has taken the work of researchers and generated a report for design or attribution of cause and effect, this requirement embodies many of the above comments. It also embodies the standards discussion since the audience expects certain standards to be met. The first and most important attribute of the finished paper is that the methodology, the evidence, the verification of the evidence EXPLAINS the cost. If a pump is needed, providing the specifications of 45 GPM at 45 PSI discharge, and what it does, “need it for the already existing stripper”, will be acceptable. However, take a plant expansion. The numbers are $1.5 million on basins, strippers, blowers, pumps, filterpresses, etc. Each unit and each piece of equipment will have supplied information verifying what the specifications are, and that it will meet specifications. For a pump, a simple purchase order with the specifications citing a drawing is sufficient. For treatment expansion, a 45 page document with details at each and every unit, Process and Information Diagram Drawings, basis for design (Height of Transfer Units (HTU’s) for stripper, gpm/ft^2 for clarifiers, etc) in standard units for that unit operation, the equations and source for the HTU’s, the settling rate and computations for the gpm/ft^2 number, etc. This is the package that is called a deliverable. It has to be physically delivered. The drawings, have to be there, the math verified, and the sources for determination quoted and checked. Most often standard engineering methods are the source. Each profession typically has several of these books. Then it goes to a public agency that audits it.
When including research, the write-up justifies the use. Most often there are several slightly different, to widely different results that will be explained. The explanation may be “I took the average”; however, it must be justified. It maybe and most often is, I repeated the research and confirmed the numbers, and use the most appropriate minimum, maximum, or the average. The use is justified. The rule is you don’t get half-credit if the bridge falls down. On anything more complicated than replacement-in-kind, very often a Professional Engineer has to sign off under penalty of law (5 years is common) that the numbers, assumptions, etc., have been verified and the design validated. This then goes to be reviewed by an independent professional engineer. With reasearch work incorporated into a design, the actual lab books and data are often requested (actually a demand, since not providing information is reason for rejection) for professional reveiw. Often questions as to different criteria or basis are asked, and if not in the literature, additional experimentation to resolve the issue may be required. More often than not, the third party must provide an auditable report that can be reveiwed by a certification board that the proposal and its review met the standards expected for the profession. This is generally required to be kept for 5 to 7 years, or even the life of the installation in case something happens and determination of the cause for failure becomes necessary. Imprisonment and monetary (individual and corporate) damages can be result of the investigation if negligence is shown. “Negligence is whatever I can convince the jury it is”, said the lawyer hired by a professional organisation training us on just what can happen when professionals sign off on a “deliverable.”
As the cost goes up, or danger to employees or the public, one can take where I used the words “”typically” or “generally”” and change to “will be””; one can take the “maybe” and change to “will be”; etc. etc.
I have found the most condemnable action, or lack of, is the readability of the IPCC. If all they were doing was providing general information, I could accept it. However, this is supposed to be a “”deliverable”” to policymakers. And I assume, that the readers have noted that the EPA relied on the 4AR as such. It has been pointed out to the EPA that the AR4 may not meet the EPA’s standards. The side by side analysis done by Steve indicates why there may be questions asked. This is not a minor consideration. The few times that the EPA has lost in court, most often have hinged on, literally, whether or not the EPA met a requirement or standard that the lawmakers put in the law. Steve has shown in his write-up what has actually been done needs to be reveiwed.
Well, ask and you shall receive. Just clicked over to wattsupwith that where the first post said there were 450 lead authors, 800 contributors and 2,500 expert reviewers. So it seems Mr. McIntyre is indeed, besides being a very funny guy when he wants to, a Nobel Prize sharer.
Click more, receive more. The Utah newspaper article about the reviewer says:
‘David Randall, an atmospheric scientist at Colorado State University and a coordinating lead author for the IPCC, said Thursday it is inaccurate to call any of its members Nobel winners. “It is a fact that the prize was awarded to the IPCC organization, not to individuals. I would never call myself a Nobel Peace Prize winner.” ‘
Maybe he wouldn’t, but the lead author whose lecture I attended did, and had fancy embossed, color certificates identifying himself as a Nobel Prize winner, which he handed out, although only to pretty girls, so I didn’t get one.
The whole thing about the Nobel Peace Prize is a shocking abuse of what the Nobel Prizes were all about. The Peace Prize is an unfunny joke of a Prize and should be scrapped.
Even worse is the abuse of numbers to claim that because there were supposedly 2500 reviewers of the 4AR this gives credence to the claims of scientific consensus and thoroughness that the actual report does not justify. Its clear that many reviewers had serious fundamental criticisms of statements made in various reports, which were simply dismissed out of hand by the authors.
I find myself wondering where Dr Bony finds herself in complete agreement with the work she supposedly signed off as a Lead Author- I’m not betting on it.
As far as “engineering quality” reports go, most climate scientists have no clue what that means. I’ll define it for you – “engineering quality” means that large amounts of money are to be invested to make a bridge and the modelling of how the bridge performs under load had better be well done and reproducible otherwise the collapse of the bridge will lead to loss of reputation and possible collapse of the company building the bridge.
Here we have a much worse situation where the explanations are little more than handwaving and spurious appeals to authority, the sunk cost as well as the future cost threatens the economies of many nations and the futures of many millions because climate scientists don’t know how to build a bridge of data, modelling and clear scientific exposition that will bear the weight of even one middle-aged Canadian statistical analyst jumping up and down on it.
Good comment John, I work with engineers and they haven’t got time for people making thing complicated, and they listen to any informed opinion because they know mistakes cost a lot of money. They also focus on risk and where exactly its greatest, because that’s where the costs and the profits are.
Thank you, John A and David S. As an engineer, my impressiom is that the difference between what we would be expected to produce and what passes as acceptable for the IPCC is dramatic. Like it or not, RW, the difference is accountability. Peer review for engineering papers is ruthless (at least as far as ASCE is concerned).
The ASCE process is, however, pretty gentle compared to the process applied for project financing and approval. I think that what commenters are really referring to when they talk about “engineering standards” is the absolute need to respond to and satisfy all commenters. The concept of blowing off a serious commenter (a la Mann et al) would be totally unacceptable in the engineering world.
I have a serious question: how many people are currently involved in climate “research”.
Ancilliary questions:
how many of them are qualified for this work (needs a definition of qualified)
how many of them are working to a pre-determined agenda
I could go on and on, but, as an engineer, I have work to do
Re: jlc (#82),
I disagree with this aspect of the comment. There are a lot of differences and handling comments is one small part of the difference.
As a start, the documents look different. For example, engineering reports are very long; they are copiously documented. Everything has to be crosschecked and verfied. Climate articles tend to be short little articles in natureorscience, poorly documented, with negligible crosschecking or verification. Climate articles typically lead to a daisy chain of references to the peerreviewedliterature, often to their own prior articles. There are lots and lots of differences – there is no need to reduce the differences to what people are “really” talking about.
Re: jlc (#82), What is really meant, I think, at risk of editorializing, when we say not engineering standard, is a failure to develop and enforce standards of research across the field. Steve’s “thesis” seems to be this, my position since my PhD has been that failure of models is invariably because of deficiency in the execution of basic principles of model development, not because of intrinsic uncertainty or complexity. Plimers thesis is this too, that climate research is ‘sloppy’ compared with other.
Researchers in other areas have worked hard to set up standards for guidance. Consider medicine as an example. Its the role of the senior researchers to set these standards.
Steve: No. My point is entirely different – even if the standards of academic research were meticulous, engineering reports look different.
I judge that most participants here would ideally like to see an engineering type report/exposition used in the communications of the IPCC on climate change and many have offerred their ideal approaches.
My question is then: How many of you think that the IPCC will recogonize the need for changing their reporting methods to anything approximating the ideals revealed here – or implement these methods given any recognition and acknowledgment of the weaknesses of their current approach?
I would guess that the answer would be resounding, no. Given that the IPCC will not be changing their methods any time soon would not the second best remedy be to point out for all to see what the IPCC is really doing and that in that situation thinking persons must take it upon themselves to determine what the “complete”, or at least more complete picture is.
I understand “engineering quality” to mean that one expects a strong, nuanced, detailed connection between what the report claims, and what actually, testably, is the situation. I expect the report to be accurate – or to assess, accurately, its inability to be accurate. I expect it to explore all challenges to its thesis. I expect it to make transparently clear the basic issues, and to lay out the evidence. I expect clear evidence for issues like “can an increase of CO2 in the atmosphere actually raise global temperatures”. I expect no area where it effectively says “everybody already knows…” without explanation. I expect it to be written in the language of an intelligent layman, using the “crystal mark” level of excellence of language, providing summaries of short, manageable length, which actually summarize the longer sections without adding extra opinions or “guidelines”. I suspect that almost all the terms like “certain”, “reasonably certain” etc should be abolished and replaced with far more evidence-based statements. I expect to find far more doubts alongside beliefs. I expect to find all temperature records easily available. I expect to find all adjustments to such records clearly explained and justified, transparently, with room for further criticism and modification. I expect a Summary of Science, not a Summary for Policymakers. And I expect it to be written after the science has been collated, not before.
.
I expect to find no room for alarmism to even begin.
.
If we cannot find “engineering quality” elsewhere, it would be good to develop it ourselves, to show, engineering-style, that it’s possible. Personally I’d like to see an “engineering quality” Climate Science wiki that meets all the above standards. And personally I think the climate skeptic community is capable of this.
.
It would be a good channelling of energy.
Steve: For the n-th time, the term “engineering quality” is not meant as a term of disapproval, but to describe a different class of report.
Thanks, Steve – it almost sounds as if you are agreeing with me: at least as far as what should be.
Any acceptable engineering report (there are a lot that are either marginal or unacceptable) will address all identifiable and feasible objections to the proposed course of action in a courteous and technically professional manner in the original report/paper itself.
Re: jlc (#87), Acceptable engineering reports do not address ALL identifiable and feasible objections. Steve is correct. A report addresses identifiable constraints and determines feasibility. Now in this sentence feasibility is broad, as is identifiable. One can determine the feasibility that something can be done. One can take that report and do the feasibility about making profits. One can take the profit feasibility and do the location feasibility to where the most profits can be made. What do they have in common: the basis, the groundwork, the argument (equations) and the goal (conclusion).
Steve is right about engineering reports do look different: meaning they read different. Try reading the economic analysis of AR4. It is a waste of time. Why? The real basis is not brought to the front. In an engineering report common practice is to highlight by seperation or even bolding the basis and equations. In AR4 one has to try to figure just what the basis and the equations used for determining Africa’s cost of decatherms as projected. In an engineering report, they are highlighted and stated with assumptions. In AR4, one will spend as much time reading why an assumption may or may not be relevant, even before finding out whether or not it was used. Why bother? If you use it; justify it. This is not to limit different possible assumptions or paths forward. It is rather a required milestone: justification.
It is the economy of words and transparency of the report that are so different. My daughter has just obtained her engineering degree from the same school and even some of the same professors I had. I gave her my books, and she did some of the same problems as I did. She learned the economy of writing engineering text as I did. Engineers do not go invent or claim some new statistic unless they can back it up. Or they lose standing. Whether the problem is new or old, the basis is stated, assumptions are justified AND identified, derivations and equations stated and then the work is justified and results validated. The answer is methodology. The methodology of the engineer includes the attribute that an engineering report can be audited, or verified and validated, and it is readable.
Engineers call it effective communication.
Steve: Once again, using terms like “effective communication” is very unhelpful in making the distinction. Many science articles are effectively communicated, but that doesn’t make them engineering reports.
Folks, if you’ve not read or browsed through an engineering report or feasibility study with at least $5 million in engineering costs, please do not opine on what you think “engineering quality” is or should be. There are too many well-intentioned but distracting opinions and suggestions.
If you have, then your comments would be helpful.
Re: Steve McIntyre (#89), Hehe. I did the justification for a $7 million vertical integration (which is currently underway) and I can second that approximate figure. There’s a massive difference in the required documentation between $1-2 million and less and the $5 million and up. It took me four years to get the project approved.
Re: Steve McIntyre (#89),
I saw the documents for a major engineering project printed out and placed on the credenza of vice presisnent in overall charge. The documents were bound together in a device similar to the ones that are used to hold the catalogs in auto-supply and hardware stores.
The shelf of documents was 6 to 8 feet (2 to 2.5 meters) thick.
I know this strays into policy areas, but since it is not an opinion on any particular policy, nor an advocacy of any particular policy, I hope this is acceptable given blog rules.
Re: John F. Pittman (#88), Very good John. The difference is focus. A science paper focuses on ideas and concepts, while an engineering paper focuses on delivering a product or service. IPCC seems to be summarising the science quite well – at least in terms of it’s basis for existence – and leaving the engineering to others. That’s obvious when there’s a Summary for Policymakers.
It’s the policy makers that need the engineering quality report, because they have to create a budget to make things happen, and try modelling various scenarios to decide what’s feasable given the constraints – the very definition of the sort of problem an engineering quality report is designed to help with. IPCC AR’s are not specific enough, not detailed enough, not audited sufficiently, and not accountable enough to provide the sort of report people here are asking for. Nor were they ever designed to, and neither will they provide such in the future without massive change to IPCC procedures. It is debatable whether IPCC should even try to produce such reports. Yet someone must and will – if only to justify why they can’t meet political goals within the given constraints.
This is the most important post I have read about “climate change”.
It deserves a much broader audience.
I strongly urge those of you who have access to the media and or to the relevant politicans and other policy makers to ensure that this is drawn to their attention and is well understood by them.
Otherwise, we will add climate politics caused economic disaster to the impact of the current global financial crash. The latter is (relatively) short term, the former, I’m not so sure.
Steve, please do not snip. It’s just too important.
Sorry, Steve, I was trying to help, to expand the picture of “engineering quality” which to me describes so much of what is missing from the IPCC science. I’m puzzled if you saw me using it as a term of disapproval because I understood and agreed with your usage. Perhaps I need to lurk. But perhaps the phrase needs more definition here, because I think it’s also going to be used for its evocative quality, whatever definition it gets. I agree with Ausie Dan, this is a crucial issue in Climate Science.
Sorrry Steve, I did not mean that effective communication is the difference. I would say that in reference to the quotes you have posted that there are basis and justification arguments missing that would be required for an acceptable engineering study at far less O&E cost than $5 million.
Steve, If it is consolation, as you know, when a mineral exploration team (mostly scientists) develops a discovery to the point of mining feasibility, the umbilical cord id cut and the engineers take over. The scientists are can be invited to browse the engineering reports and they have some idea of what is involved. So some people understand your distinction.
There is is also some exchange. I once managed a pilot plant as a scientist, although the work was almost all engineering. The engineering reports had to be comprehensive and good because we were using 10 tons a day of chlorine gas at 1,050 deg C. A failure would have affected the town around use. Perhaps severely. Maybe for reasons of background, I am as critical as you are about standards of reporting.
Re: Geoff Sherrington (#96),
Geoff
The umbilical cord is not completely cut, actually – the engineering studies still have to reflect the geological modelling: such studies cannot ignore the geological constraints, much as some engineers may wish them to.
I’ve had geological input and consequent due diligence review of many projects globally (not blowing a trumpet, just a fact) with funding to AUD$1bn. Since geological modelling has a degree of risk (we cannot know 100% the full state of any deposit – at least not until it’s fully mined out), the engineering costs must reflect that risk in a realistic manner. Such risks also include potential upsides.
In my experience, the tension-filled combination of geo-scientists and engineers is about the most productive I’ve ever encountered, precisely because those risks are recognised at all stages. The AGW science cohort seems to me not to recognise any of the risks associated with inadequate modelling, at least not in public.
I find that all the excellent contributions above on the subject of professional standards and the parallel observations that IPCC reports fall short of meeting recognized specifications should be cause for concern at the level of UN-member national governments.
In general the leadership that has forced international agreements on emissions controls has come from professions that have no such standards. The obvious reluctance of governments to adopt another Kyoto-like set of emissions constraints based on the justification presented in the IPCC reports is a sign that the IPCC’s arbitrary standards and practices are just not good enough.
The abuses of climate science by NGO opportunists and some government agencies also makes the case for both “engineering” and ethical standards.
When legitimate attempts to analyze and control the Earth’s atmosphere become confused with fundraisers’ profiteering we are all being gored.
So Team, what’s the next step?
Graeme, the next step is the same as the last few. Completely ignore this work by Steve and others and continue to claim that the sky is falling. Why change a winning game ?
Can anybody out there improve upon James Erlandson (#64)‘s example? That is, can anyone suggest some specific example of a ‘proper’ engineering report [a la Steve McIntyre (#89)] that is publicly available for browsing through?
Apparently there is a new study in Science concerning clouds and feedback. I just saw this story at Bloomberg.
I just read this article, it was linked to from WUWT. It would seem that daytime cloud cover is important to the cooling effect of clouds, while nighttime cloud cover would be important as a warming effect. It would also seem that the variation between during which time clouds are dominant should be given a thorough inspection.
It won’t be evenly split between day and night, and I would think it strange if the clouds stayed dominant in only one daytime phase. The most difficult part of this analysis would be coming up with the global average data to compare with the global average temperature, imho.
Oops, strike “during” in my above comment. Sorry, didn’t finish deleting. 🙂
3 Trackbacks
[…] thought this article by Steve McIntyre was an interesting window on the IPCC process. Frequent readers of this site know that I believe […]
[…] Read the entire Climate Audit post: ‘Boundary Layer Clouds: IPCC Bowdlerizes Bony’ […]
[…] Of course, the removal of any doubt about man-made global warming from the IPCC reports seems to be fairly clear. Interestingly, the language in IPCC AR4 is (using the terminology of climate science) […]