IPCC: AR4 guidance on uncertainty

John A writes: I thought this would be interesting to note in passing. The IPCC has “Guidance Notes for Lead Authors of the
IPCC Fourth Assessment Report on Addressing Uncertainties
“.

Amongst its recommendations:

Make expert judgments

5. Be prepared to make expert judgments and explain those by providing a traceable account of the steps used to arrive at estimates of uncertainty or confidence for key findings — e.g. an agreed hierarchy of information, standards of evidence applied, approaches to combining or reconciling multiple lines of evidence, and explanation of critical factors.

6. Be aware of a tendency for a group to converge on an expressed view and become overconfident in it [3]. Views and estimates can also become anchored on previous versions or values to a greater extent than is justified. Recognize when individual views are adjusting as a result of group interactions and allow adequate time for such changes in viewpoint to be reviewed.

Be aware of a tendency for a group to converge on an expressed view and become overconfident in it“? Imagine that.


18 Comments

  1. Jos Verhulst
    Posted Jun 10, 2007 at 3:45 AM | Permalink | Reply

    The conclusions presented by the IPCC depend crucially on the practice of ‘expert judgement’ (see for instance:“The approaches used in detection and attribution research (…) can not fully account for all uncertainties, and thus ultimately expert judgement is used to estimate the likelihood that a specific cause is responsible for a given climate change. The approach in making judgements (…) is to assess results from multiple studies using a variety of models, forcings, analysis techniques, and observational data sets and to subsequently conservatively assess the likelihood of the hypothesized link in a likelihood level below the consensus level of significance inferred by the different studies. Such a decrease in significance levels by expert judgement attempts to account for remaining uncertainties that are not accounted for, such as limited range of exploration of possible forcing histories of uncertain forcings, or structural uncertainties” FAR, chapter 9, 1.2). Personally, I have never understood how ‘expert judgement’ can play such a role in science. I suppose that a scientist can, when he has made a lot a related observations, produce an educated guess concerning the outcome of the next observation. But without an actual observation, such a guess could not be called ‘scientific’ in the usual sense of that world.

    There seem to be two problems with the use of ‘expert judgement’. Firstly, when the ‘expert judgement’ does not concern the outcome of what would be a routine observation, it seems unclear how it could be of any value. Secondly, even when the ‘expert judgement’ concerns a routine observation that has not actually been made, it should not be called ‘scientific’, although it could be of some value in jurisdiction, for example.

    So there are two separate questions to be answered:

    => is ‘expert judgement’ used by the IPCC only for routine observations, where an extrapolation based on the outcome of actual observations has a reasonable chance to be true?

    => is the outcome of a mere ‘expert judgement’ presented as scientific fact instead of just a guess (although an educated one)?

  2. Reference
    Posted Jun 10, 2007 at 5:22 AM | Permalink | Reply

    The usual term for this type of epistemological distinction is certainty. By using the terms confidence and likelihood perhaps the author is attempting to distinguish between the certainty of subjective and objective knowledge. Popper discusses all this and much more in his book ‘Objective Knowledge’ (OUP 1979)

    As an aside, Popper defines the method of science as ‘bold conjectures and ingenious and severe attempts to refute them’

  3. Murray Duffin
    Posted Jun 10, 2007 at 6:38 AM | Permalink | Reply

    :“The approaches used in detection and attribution research (…) can not fully account for all uncertainties, and thus ultimately expert judgement is used to estimate the likelihood that a specific cause is responsible for a given climate change.
    As a non-scientist I can see some value in using this approach if all reasonable specific causes are reviewed holistically and with the same degree of expertise. (We do this a lot in addressing engineering problems whereb the root cause(s) are not proximate to the effect). When experts working with deep knowledge in a narrow field reject possible causes outside their field of expertise they are unconsciously applying inexpert judgment prior to getting to their application of expert judgment. When this problem is aggravated by the human tendency to paradigm paralysis the whole idea of expert judgment becomes very questionable. The IPCC seems to suffer strongly from this problem in their consistent failure to consider solar forcing in any terms except irradiance.

  4. Stan Palmer
    Posted Jun 10, 2007 at 6:56 AM | Permalink | Reply

    Terms such as “satisficing”, “certainty” and “truth value” are used in fuzzy analysis. Expert judgements are used to determine these since, commonly’ no other means are available. They are used in much the same way that”confidence” is used in the table above. Fuzzy techniques are used to create working controllers.

  5. Ken Fritsch
    Posted Jun 10, 2007 at 11:49 AM | Permalink | Reply

    The concepts and words describing them mean nothing to the reasonably observant reader, who has actually seen these methods applied and misapplied, without having the capability to examine how the process of determining uncertainty and likelihood was actually handled by the individual working AR4 groups and who specifically made the expert judgments. Traceable accounts would seem to imply that those documents are the critical factor in doing a proper examination. It was those documents that I requested from the IPCC.

    Remember Hansen, et al. have waxed eloquent on the need for more quality control of weather (temperature) station measurements.

  6. RomanM
    Posted Jun 10, 2007 at 12:06 PM | Permalink | Reply

    #4

    What seems to have been done by the authors of the IPCC report is not science. Their “calibrated levels of confidence” in Table3 of the guidance notes represent five categories of opinion: pretty sure, maybe, haven’t a clue, maybe not, and pretty sure not. The numerical “chances” associated with them are spurious. How exactly do you decide that some statement has “about a 7.3 out of 10 chance of being true?” Maybe they did an opinion survey of their membership. Then they “calibrated” the responses on a standard set of “correct” answers. Using principal components, it would have been trivial for one of their “statisticians” to determine that “the probability that statement 17 is true” is .73 with a standard error of .02. The numbers are “virtually certain” to be window dressing to justify the “likelihood” levels of Table 4 since there does not appear to be any explanation or justification given in the report for any of their likelihood statements.

    Another nice touch is to not indicate what part if any of any their “probability” conclusions is based on an actual study or just on personal opinion.

    RomanM

  7. John A
    Posted Jun 10, 2007 at 3:43 PM | Permalink | Reply

    I would guess that it would be more a measure of the “expert reviewers” confidence in his or herself. I think its pernicious and dangerous for reviewers to be making expressions of statistical confidence and likelihood about things other than the results of experiments.

  8. Posted Jun 10, 2007 at 5:55 PM | Permalink | Reply

    I noticed the careful grammar and parsing of it in the IPCC publications, and figured there was a methodology behind it. The difficulty or necessity of this type of top down structure can be argued, but I would prefer to hear individual non-predigested attitudes and opinions.

  9. Ken Fritsch
    Posted Jun 11, 2007 at 10:12 AM | Permalink | Reply

    I would guess that it would be more a measure of the “expert reviewers” confidence in his or herself. I think its pernicious and dangerous for reviewers to be making expressions of statistical confidence and likelihood about things other than the results of experiments.

    I would guess that the IPCC considers it a good marketing strategy whereby they can impart a show of more objectivity mixed with a wider spread of expert opinion than might actually exist in the eyes of a beholder of all the details of how the methodology was really applied.

    I am surprised that there have not been at least a few leaks from people exposed to the process on how it actually worked. They could not all be true believers — could they?

  10. jae
    Posted Jun 11, 2007 at 10:56 AM | Permalink | Reply

    “tendency for a group to converge on an expressed view and become overconfident in it.” LOL. Isn’t this the whole problem with the “consensus” argument?

  11. Posted Jun 11, 2007 at 12:52 PM | Permalink | Reply

    providing a traceable account of the steps used to arrive at estimates of uncertainty or confidence for key findings

    So where is the instrument calibration record traceable to NIST?

  12. RomanM
    Posted Jun 11, 2007 at 2:48 PM | Permalink | Reply

    #7

    JohnA, you are quite correct. This IS really an indication of the level of personal confidence (in a non-statistical sense) with which an opinion is held. There might not be an issue if the numbers were phrased as “On a scale of 0 to 100, how strongly do you feel that statement X is true?” since this could obviously be understood as an opinion of an opinion. As it is presented in the IPCC document, it is purported to be a scientific numerical summary about the truth of the statement itself. When it is put into such probabilistic language, the reader views statement X as the outcome of some sort of random experiment and we all know how often events with a 99% probability occur! Manipulation of the finest quality. On the other hand, if the people at IPCC genuinely believe that these are meaningful “probabilities” or “confidence levels”, with their interpretation of them, they are demonstrating the same statistical naivety of those who believe that a 95% confidence interval means “the probability that the parameter you are estimating is actually in the interval you just calculated is 95%” as opposed to “the random procedure that leads to the interval is correct about 95% of the time.” There are obviously no such random procedures involved in the conclusions they put forward and their “likelihoods” are spurious.

    What makes this more dangerous is that, with the apparent lack of some sort of general procedures, the actually “consensus” decisions about the “likelihoods” would be made by a relatively small group of people ‘€” the committee of authors of the actual report. I can just see this handful of people sitting around a table voting on the motion that “a likelihood of 92% be assigned to statement X.” In this group of zealous advocates of AGW, you know how the vote will turn out.

    RomanM

  13. Joe Ellebracht
    Posted Jun 11, 2007 at 6:32 PM | Permalink | Reply

    Let me just state here, for the record, that A Gore, Jr. has a bigger ego than Michael Moore (medium confidence).

  14. Ken Fritsch
    Posted Jun 11, 2007 at 8:08 PM | Permalink | Reply

    Re: #13

    I can just see this handful of people sitting around a table voting on the motion that “a likelihood of 92% be assigned to statement X.” In this group of zealous advocates of AGW, you know how the vote will turn out.

    Since the IPCC has not revealed the working groups/chapters methods of determining likelihood, one could hypothetically conjecture a situation where 5 scientists voted for a likelihood of 95% for an event and 4 scientists voted against that level and so the reported likelihood gets a worded version of 95%. What if the 4 dissenting scientists thought the likelihood for that event was 50% — or less?

  15. RomanM
    Posted Jun 12, 2007 at 6:09 AM | Permalink | Reply

    The major objection I have with their approach is that there are no legitimate methods of making genuine probability statements in traditional probability theory about the truth of a statement concerning reality. People listening to what they are saying will use their guidelines to interpret the numbers as meaningful scientific results as opposed to what they actually are – some unknown persons’ opinions of self-held opinions. If IPCC is using some sort of fuzzy probability methodology, then certainly the world is entitled to know what they did and how it was done. Unfortunately, my suspicion is that there is no scientific process involved and their traditional lack of transparency serves to hide that fact.

    RomanM

  16. Chris Wright
    Posted Jun 12, 2007 at 11:25 AM | Permalink | Reply

    In April New Scientist ran an iteresting article about the psychology of groups. ‘Group think’ is actually a technical term that refers to the state of mind that can lead to terrible results, even, in extreme cases, torture. I believe that, possibly aware of an obvious thought, in an editorial they stated that the IPCC was not a victim of group think.

    I’m not so sure. Bearing in mind how the IPCC has mutated over the past twenty years, and how it operates, it may just be that the IPCC is a text book case of group think.

    A quote from the article’s header:
    “No matter how free-thinking you believe yourself to be, says Michael Bond, being part of a group can make you do stupid or even despicable things”.

    Does that sound familiar?

  17. Steve Sadlov
    Posted Jun 12, 2007 at 12:25 PM | Permalink | Reply

    RE: #17 – Groupthink led to the self imposed reversal in progress from 1968 – ?

    It opened the door to anxiety driven bad decisions based on the flawed premise that Man was on the verge of various “tipping points” – population, oil, nuclear war, climate, etc. This then led to anxiety driven rather than anticipation driven views regarding the future.

    Take for example the Space Shuttle debacle. That was an expression of the notion that the space program must abandon manned exploration in favor of a LEO based program to create near Earth “space industry” which was touted to be “for the benefit of the masses” and somehow more morally correct than manned explortation, which by 1968 had become very uncool – very “imperialistic” and very tied into the so called “military industrial complex.” So now, we have a lemon which is a roll of the dice of the astronaut’s lives every time it is launched. Imagine if instead we’d continued doing what we were doing and never stopped following the original 1960-61 plan? We never would have lost the 30 plus years we’ve lost, not to mention all those valuable talented people who’ve perished needlessly owing to O-rings and perforated heat shields.

  18. kp
    Posted Jun 12, 2007 at 3:47 PM | Permalink | Reply

    It seems to me that AGW is more of a philosophical movement than a science validated cause/effect. Why else would such psycologically subversive methods be chosen as a medium to present the world with a “well balanced” scientific assesment?

    It doesn’t require much precience (or a degree in social psycology) to predict that legions of frenzied AGW believers will point to this “method” of verification as the pinnacle of scientific objevtivity. Does it seem to anyone else that this is a rather Post-Modern approach to forming conclusions?

Post a Comment

Required fields are marked *

*
*

Follow

Get every new post delivered to your Inbox.

Join 3,115 other followers

%d bloggers like this: