Julien Emile-Geay from Judith Curry’s university – who, together with Kim Cobb, is teaching a course on the Hockey Stick – has joined our debate with a forceful criticism of Craig Loehle’s recent paper. While Emile-Geay seems to be a lively young man with some very cordial comments about CA here and his comments are very welcome here, his initial skate around the hockey rink seems surprisingly unreflective about the defects of the canonical Hockey Team studies. Whatever validity his points have against Loehle, all too often they apply even more forcefully against Team articles, none of which are criticized. A little more attention, shall we say, to the “beam in his own eye” or at least of his teammates.
Emile-Geay asks of Loehle:
Where are the CE, RE, and most importantly R-squared statistics that are so dear to ClimateAuditers ? How are we supposed to guess whether the reconstruction has any skill ?
I agree with this 100%. These statistics are part of the game and should be provided. While I think that these statistics have to be very carefully assessed and that the risk of spurious RE statistics is not understood by climate scientists at all, I agree that readers are entitled to such information about any proposed reconstruction presented as a positive alternative.
But the more interesting issue in this demand is surely not the performance of the Loehle reconstruction, but the dissonance between Emile-Geay’s demand for a verification r2 statistic from Loehle as compared to past contortions by Mann (and Ammann) in trying to cover up the MBH verification r2 failure.
As we speak in November 2007, Mann has never reported the verification r2 (or CE) statistics for any MBH98-99 steps prior to the AD1820 splice. You can confirm this by examining the original MBH98 SI where the RE is reported but not the verification r2. On the other hand, MBH98 Figure 3 shows a map reporting the AD1820 verification r2 results – a step for which results were favorable, unlike the earlier AD1400 and AD1000 steps. MBH98 also explicitly says that verification r2 results were considered and IPCC TAR states that the Mann reconstruction had “skill” in verification statistics without limiting this to the RE statistic.
Mann has never actually reported the reconstructions for the individual steps, including the AD1400 or AD1000 steps, forcing any interested readers to run the gauntlet of trying to replicate his study from scratch in order to obtain the elementary statistics said here by Emile-Geay to be a necessity for any study (and a demand with which I agree.) I have been trying to obtain Mann’s actual result (or equivalently the residual series) for the AD1400 step since 2003 without any success in order to carry out the verification tests demanded here by Emile-Geay. In 2003, Mann refused. I asked the National Science Foundation and they refused. I filed a Materials Complaint to Nature and they refused, saying that Mann was not required to produce the results of his “experiments”; it was up to him.
In 2004, I acted as a reviewer for a submission by Mann to Climatic Change, supposedly rebutting our EE 2003 article. In my capacity as a reviewer, I again asked for this information and again Mann refused. Ultiumately Mann withdrew the article rather than providing the data (although the rejected article is check kited in Jones and Mann 2004.)
In MM(2005 GRL and 2005 EE), we observed that the verification r2 for the AD1400 step under consideration there was ~0. In MM (2005 EE), we observed that it seemed inconceivable that Mann had not calculated the verification r2 statistic (as indeed he had as evidence by his source code).
This issue has prompted much subsequent controversy. The House Energy and Commerce Committee asked Mann, Bradley and Hughes whether they had calculated a verification r2 statistic and asked what it was. Even in response to a congressional inquiry, Mann refused to provide the verification r2 statistic. He did provide code and the code shows for certain that he calculated a verification r2 statistic in the same source code step as the RE statistic – observed in summer 2005 at CA.
The National Academy of Sciences specifically referred to the verification r2 issue when they wrote their complaint letter to the House Energy and Commerce Committee. In our presentation to the NAS panel, we summarized the verification r2 issue as it then stood. In the NAS panel presentations, a NAS panelist asked Mann whether he had calculated a verification r2 statistic for the AD1400 step and what was it. Mann famously denied calculating it, saying that that would be a “foolish and incorrect thing” to do – notwithstanding the fact that his own source code, produced for the House Energy and Commerce Committee and the MBH98 figure for the AD1820 step, showed that he had calculated the statistic: he merely didn’t report it for the adverse steps.
As an anonymous reviewer of Wahl and Ammann, I asked the authors to report the verification r2 statistic for their calculations (fully knowing that the results were zero). They refused, while, at they same time, they issued a press release stating that all our claims were “unfounded”. I’ve reported previously on my proposal to Ammann in San Francisco in Dec 2005, proposing a joint paper itemizing points that we agreed on, points that we disagreed on, knowing that our codes fully reconciled (while neither fully reconciled to Mann’s); he said that this would interfere with his career advancement. I also urged Ammann to report the verification r2 results telling him that he seemed like a nice young man but that I would not stand idly by if he failed to report the adverse results; he still said that he would not report the verification r2 statistiis. I filed an academic misconduct complaint and, in late February 2006, the adverse verification r2 results were disclosed in an Appendix to the revised Wahl and Ammann, fully confirming our previous results (although Wahl and Ammann did not credit us with priority or acknowledge that they had confirmed our earlier findings).
The Wahl and Ammann preprint came online a couple of days after the NAS panel hearings. In a supplementary letter to the NAS panel, we alerted the NAS panel that the revised Wahl and Ammann had confirmed the adverse verification r2 scores, and the NAS panel noted these failed verification r2 results in their report.
While I fully agree that Loehle should have reported the verification r2 statistics for his reconstruction (and I would be surprised if they were any better than the results for MBH or other Team studies), it is extremely hypocritical (and all too characteristic of Team climate science) for Emile-Geay to criticize Loehle for this omission given the history of obstruction on this matter by Mann and Ammann. If Mann wouldn’t provide this information to the NAS panel even when asked directly, shouldn’t that (and related ) refusals have occasioned Emile-Geay’s disapproval long before his opprobrium against Loehle’s omission of this statistic (an omission which should be corrected).
BTW I’ve calculated the Loehle performance statistic relied on exclusively in Juckes et al 2007 – the 1856-1980 r. Loehle’s recon has an 1856-1980 r of 0.594, which matches or exceeds the 1856-1980 r reported by Juckes for several CVM variations (MBH .535, Esper 0.599, Jones et al 1998 0.367). It is my view (expressed in my review of Juckes et al ) that his claim that these correlations were 99.99% significant was absurd, because trivial variations with different medieval-modern relations also had 99.99% significant 1856-1980 correlation (now including the Loehle reconstruction). Juckes was not required to even respond to that criticism (in breach of Climate of the Past policies), but the issue is surely back on the table.
Climate Audit 
233 Comments
Not just that, providing false and/or misleading statements to Congress is a serious matter. Were they under oath?
#1. Mann’s statement was to the NAS panel and was not under oath. The house committee lost interest in him. After Mann’s presentation, as an observer, I said to the panel that I thought that they had neglected their duties in not resolving this matter. Mann walked out when I started speaking (thereby avoiding any follow up). The two statisticians sat like bumps on a log when Mann made this absurd statement. Obviously JEG doesn’t think that a verification r2 statistic is “foolish and incorrect” since he criticizes Loehle for not calculating it.
Cutting stuff Steve. A clear and documented expose of the double standards of “Climate Science”.
I look forward to JEG’s reply.
I would be surprised to see JEG back on this blog. Which is a pity because obviously he is in a position to reveal plenty.
I personally do not believe the Mannian “error analysis” is worth anything. However, since JEG seems to be insisting on those, I calculated the Mannian “CI”s for the Loehle reconstruction the following way:
1) I took the HadCRU global instrumental series and 30-year run mean filtered it (in order the target series to match the Loehle reconstruction)
2) stardardized the both series to the mean of 1864-1980
3) calculated RMSE (over the overlap 1864-1980) between the series (which gives the Mannian CI sigma).
I sent my files to UC for double checking, but here are the preliminary results:
RMSE=0.067, so that gives Mannian CIs as 2*sigma=0.13! BTW, R2=0.73 and the series are “remarkably similar” using the Mannian terminology.
There’s some “skill” for you, JEG. Have fun!
JEG took up valuable bandwidth that could have, and should have, gone towards evaluating Loehle’s claims. JEG’s statements that Loehle’s work was pseudo-scientific rubbish did nothing but inflame most posters and divert them from far more profitable lines of criticism. Bender’s criticisms were cogent and directed towards improving Loehle’s study while JEG’s were purely destructive and, in my opinion, vindictive because his ox was being gored.
I look forward to Steve’s critique of Loehle because I can be reasonably sure that when its over, I will know more about the strengths and weaknesses of Loehle’s arguments.
Steve: I strongly disagree with this. I found Emile-Geay’s criticisms to be very revealing and I’m very glad that he took the opportunity to comment here and welcome more comments from him.
this argument of course goes both ways. i am not sure, whether it is useful to continue to discuss it.
the claim that other papers are “more” unsound in this aspect is a strong one and needs much more evidence.
before looking any closer i would say that papers that passed a real peer review are superior in this aspects. multiple scientist have pondered over the methods and decided to accept them and their (potential) shortfalls.
—————————–
i have a real question:
how are you going to calculate correlations or errors for 1995 in a paper, that has the data ending in 1810?
i strongly suspect that this error bar will have a MASSIVE hockey stick form…
Steve said:
“I strongly disagree with this. I found Emile-Geays criticisms to be very revealing and Im very glad that he took the opportunity to comment here and welcome more comments from him”
Let’s wait a month and see if JEG’s commentary leads to a better understanding of the issues surrounding Loehle’s work or if they only serve to divert the energy of the posters. I actually hope that you are right, it would be the best outcome, but my experience with this kind of critic makes me very skeptical.
Steve: I didn’t say who the comments were revealing about. You’re assuming I meant Loehle.
PS to #10
If I’m wrong, I say so and kick another $20 into the tip jar.
Let the force be with him.
===============
Re #9
Are you willing to wager $100 in the CA tip jar?
Re #10, Bob Meyer
I suspect you might be missing Steve’s point …
Suggestion: Rather than screaming your ‘real questions’ in bold, why don’t you just restrict yourself to real questions all the time. Then you can save the bolding for when you really need it.
To your question. The sparseness of the datasets in the time domain has been commented on more than half a dozen times already. It is recognized this is a problem, not with the paper, but with the datasets – just as it is a problem for Moberg et al., or anyone else who wants to use them.
If data are missing, sir, there is nothing you can do about it. The overlap with the instrumental period is, yes, going to be very small for a series that extends to 1810. The calibration is going to be very weak. That’s all there is to it. You make a decision what your proxy criteria are and you apply it evenly to all your datasets. Loehle did not screen proxies based on calibration significance. He documented that. You disagree with that practice. Fine. How would you do it? Would you be so selective that your network ends up covering an even smaller part of the globe? You see the problem of making opretenses at having a higher standard than everyone else? There’s not a lot of data to choose from!
Fortunately for the Loehle paper, there is only one of 17 series that is this short. Now, if only you could reduce your bandwidth for that redundant argument by 1/17th.
Judith Curry has stated:
And this statement was made after prefacing it with: Peer review guarantees nothing.
In my view the double standard some of us see very clearly in the way some climate scientists criticize these articles is hinged to this view expressed by Curry, and indirectly by JEGs failure to reference his criticism of Loehle to similar Mannian failures, that is, that peer review, while guaranteeing nothing, can be held out (even if one most go a step further and differentiate amongst peer reviewers) as a rationale for suspending vigorous analyses and attention to those that do perform extensive analyses of theses papers. A good example of this reaction is seen in the methods of the IPCC where peer review is the gospel in selecting literature to cite in support of a case for a pre-determined policy stance. If the literature is peer reviewed (and essentially supports the policy) it can be held above the fray of criticism. On the other hand, literature that goes against the pre-determined policy can either be ignored or simply dismissed as being lonely findings that are not supported by the consensus view.
Bob Meyer, [JEG]’s comments reveal much about himself.
Kenneth, JEG’s criticism of Loehle’s work is either valid or it is not. Whether JEG has applied the same standards to other works is not relevant to the validity of his current criticism, but to his standards of behavior. Complaining about a double standard does not deflect from scrutiny of the work in question or reduce Loehle’s responsibility to do the best he can and take advice and criticism where warranted. The existence of a double standard is important to recognize, yes, but the criticism is either valid or not, regardless of the critiquer or the fairness of the system.
I just now read the rest of the comments.
Good grief, when I’m stupid I am unsurpassed.
Steve, I’ll give you the 20$ if you promise not to tell anyone about this.
Good morning all. Contrary to some claims.. I am back and kicking. Though in truth, i was kicking little too hard on Friday. I apologize for a certainly inflamed language, which was destructive . Though i was upset at what i still believe to be a misleading piece of pseudo-science, this is no reason no lose one’s temper. I do science because i care about truth, and i hate to see it mistreated , but that is no excuse for anger. There never is one, and ultimately, one’s anger mostly harms oneself. So i took the opportunity to reflect over the week-end, and posted the result on my own blog.
I will welcome any scientific comments having to do with this reconstruction . Please note I am not here as the scapegoat of climatology, so do not ask me to defend the IPCC or all of the climate science literature. I hope my disclaimer in that post is clear enough – i can refine it if needed.
Answer to Bender’s many posts : though i do welcome your eagerness to reproduce Li et al (2007)’s results, please stop requesting data from me and refer to their website. I have no more access to it than you do, so let’s stay on point here. This is about Loehle.
I will make an effort to address Steve McIntyre’s well-posed questions shortly (which may mean tomorrow : i have a class to prepare today…).
I do enjoy this debate and i hope we call all get to a constructive place : i.e. as close to “truth” as possible, whatever that may be.
The pandora box is now open…
sod,
Did MBH98 “pass a real peer review,” considering there’s no way “mutliple scientist have pondered over the methods” since MBH98 didn’t fully detail the methods. It wasn’t until M&M that this was done.
Why don’t you look at MBH98 for starters, look at how many proxy datasets cover limited time periods in that publication, figure out how errors were calculated in that paper, and get back to us.
Susann,
Loehle’s data is no more flawed than Moberg et al. (2005). His lack of reporting of robust calibration statistics is no more flawed than MBH9x. The scientific community at large sets the standard for publication. What you hope for is even application of the standard.
Sorry, but you persist in this idealistic notion that scientific papers (and their reviews) are right or wrong – and that just is not how scientific publication works. Read Judth Curry’s comments; she understands well how the business works and how this serves society. No one can afford to amass a lifetime of data and publish one correct paper. Scientific progress is incremental. Publication documents progress.
JEG’s review is, substantively, no different from mine. The only difference is our editorial judgment – which is not a reviewer’s role. I judge that Loehle 2007 (with major revision) would easily meet the minimum standard for publication. JEG felt otherwise. An editor would have to look at the two reviews, and, seeing that they’re so similar in content, but so different in editorial advice, would have to make his own judgment call. That’s review, and, like it or not, that’s the material policy-makers must learn to work with.
I am more concerned about the Jones. Thompson, Mann “kimonos” than Pandora’s box.
JEG,
Thanks for participating. I had confidence you’d be back.
All reviewers react viscerally to papers when they first read them. When formulating a final reply one sometimes has to work quite a bit to take out the inflammatory language. The only mistake you made was posting before reviewing your own review.[snip]
RE 7. sometimes when your lawn gets really patchy you have to rip up the sod.
Sod on sod off.
“this argument of course goes both ways. i am not sure, whether it is useful to continue to discuss it.”
“I am not sure it is useful to discuss this” is CODE FOR : “I’m losing , can we change the topic.”
“the claim that other papers are more unsound in this aspect is a strong one and needs much more evidence.”
Well, here is a thought. Detail what would convince you? If a researcher witholds data. Good thing? or bad thing?
If a researcher hides his own calculations that put doubt to his consclusions: good thing? or Bad thing.
If a researcher continues to rely on data and methods that experts have serious issues about: good thing or bad thing.
Question: how much more evidence do you need? It is a common complaint amongst AGWers, that no amount of data
will satisfy a sceptic. Fair. So, draw the line on Pennstate mann. Proposition. His reconstruction is flawed.
what will it take to convince you.
“before looking any closer i would say that papers that passed a real peer review are superior in this aspects.
multiple scientist have pondered over the methods and decided to accept them and their (potential) shortfalls. ”
Seriously. A paper was reviewed by somebody. you dont know who. you dont know what comments they made. you dont
know ANYTHING but this: The journal has a written policy about review. You dont know if they follow it. you dont
know how they followed it. you can Observe this. they claim to have a policy. they claim to follow it.
You have no evidence that they do. no evidence that the policy works. Nothing scientific whatsoever.
You have sociological musings.
SOMEBODY told you that peer review was used. Somebody told you that peer review actually has anything to do
with truth value. You believed this.
Now, study the formation of canons.
17, I think we all understand that two wrongs don’t make a right, and I think any discussion of double standards is an observation that stands on its own, and doesn’t justify anything.
OTOH, if you’re doing things as best as is possible, and it’s still deficient, it’ll be deficient with all such studies, and then it becomes relevant that such a flaw is universal. Pointing out a double standard is the same thing as making it clear that all such studies have these deficiencies.
The correct response to that situation is to try to make it as clear as possible what the limitations of the work in question are.
Re #20 :
bender says:
November 19th, 2007 at 9:45 am
I most fully agree. That is why respectable journals always mandate at least two reviewers, and potentially more if the reviews disagree wildly. No “peer” is ever perfectly objective, which means the editors must provide extra balance. That being said, they are human beings too, and no one is infallible . This is why there will be much to be said for an “open source” review as is conducted here. I do believe in democracy and in science, and it is a fascinating experiment to see the two merge.On the other hand, this is a pandora’s box in and of itself, because no scientist has the time to deal with a zillion trolls’ comments – and that requires a filtering system which will also be undoubtedly subjective… Hence the emphasis on “peer-review”. Does anyone have suggestions for a happy middle-ground ?
Sure thing, JEG. It was actually quite unfair of me to apply that kind of pressure.
RE 18 Dr. JEG.
I apologize for kicking you in the shins. Won’t happen again. (Next time I’ll aim higher. Just kidding.)
Welcome back. Peace.
#18. Julien, thank you for your comments. I don’t think that it’s as easy as all that to find a sure basis on which one can say that Loehle is a “misleading piece of pseudo-science” without also applying that term to Moberg, MBH, Juckes and their ilk. I’ve obviously been aware for some time that one can make “apple picking reconstructions” with different modern-mediaval levels that cannot be objectively distinguished in the calibration period with elementary verification statistics.
The difference between Loehle and Moberg results is not really the use or non-use of tree ring data, but the somewhat different suite of low-frequency proxies – especially the exclusion of the questionable Arabian Sea G Bulloides and the uncalibrated non-normal Agassiz melt %. If one did a Moberg-type recon with the Loehle low-freq network and Moberg tree rings, my guess is that it would still look a lot like Loehle’s recon.
There are some things that Loehle should have done like verification statistics – stuff like that. And criticism on these counts is deserved. But that is readily curable. And once that is done, I think that it’s pretty hard to find objective criteria by which Moberg and Mann pass into the realm of “science” while Loehle is consigned to “pseudo-science”.
25, I completely agree with everything that you said. As far as your question about troll control goes, this is just a system that has to be tuned. If this particular blog gets too much noise, a registration system may be needed for commenting. There are ways to deal with that. As with anything new and novel, it’s going to have to evolve and mutate. We’ll just have to see where this goes.
Re: #17
I find a number of JEG’s criticisms are valid for any temperature reconstruction, even if not always politely made in reference to the Loehle paper. I have made a number of criticisms myself on temperature reconstructions and that includes Loehles. My point is that critical and potentially life changing policy should depend on vigorous analyses and criticism of all contributing papers, regardless of what policy they might support. That point was made in my IPCC example.
In the case of Loehles reconstruction, his cooperation with posters here is such that I think we will be able to quickly determine the amount of uncertainty (or at least how well it can be determined) in his reconstruction. Doing the same for a number of other temperature reconstructionists has appeared to be much more difficult to impossible. I suspect that a reasonably valid statistical accounting of the uncertainty of any temperature reconstruction will be large and it is that uncertainty (as opposed to a show of hands of involved scientists) and/or the failure to correctly determine that uncertainty that should be made available to policy makers and those interested parties that will have to live with the results of attempted mitigations.
What is being expressed here, Susann, is that if someone like JEG can see the weaknesses in the Loehle paper than they must see it in the Mannian type temperature reconstructions and reconstructions unlike Loehles that have been used by the IPCC and other policy promoting operations in support of immediate AGW mitigation. If these analyzers do not voice the same criticism across the board, then like those posting here who confine their criticism to Mannian reconstructions, they become suspect of being guided by other motivations.
JEG, those “trolls” make you think. You simply can not reply to the zillion of them. But they are nevertheless real people and facing them keeps the scientist grounded. The problem is the fragile academic ego who fears the troll that could embarrass them. My advice is: face your fears! Society pays you well. You owe it to them to listen to their arguments and to think. So what if you get burned once in a while; that’s what you’re paid for! Closed science is rotten, junk science, controlled by dirty internal business concerns. Open science is to the benefit of everyone.
Could somebody suggest a good textbook (or website) or two on statistics? I find that I am a clue-free zone when folks mention these things like R2 and RE stats, and principal component analysis is very hazy to me. As a result, this entire discussion is mostly going over my head.
Steve: This is a fair question, but please don’t insert on it on a topical thread. I’m going to move this to Unthreaded. Substantively there are many local methods used in Team climate science which are little discussed off the Island and the answer is not all that easy.
Jeg, followed your link and found it informative. One line in particular;
“Let me be perfectly clear. I am not here to defend previous work : the sole focus of this review is Loehle’s article.”
Yes, readily curable.
[JEG] will learn that “pseudo-science” is a serious claim that demands serious supporting argumentation. Loehle (2007) is not pseudo-science. I would even argue that it is “normal science”. This would be consistent with GRL’s lack of interest in reviewing it.
JEG says:
One of the purposes of audits is to ensure that criteria are objective and not arbitrary. So previous work is relevant to that. You can’t have one set of standards for Mann and Jones 2003 and another set of standards for Loehle 2007. That’s the type of hypocrisy that drives people crazy. If Loehle cannot meet the objective standards that underlay acceptance of Mann and Jones 2003, Moberg et al 2005, Juckes et al 2007 etc, then it’s fair enough to reject it. If he meets those standards, then it’s not fair to say that we’re applying new and more stringent standards (which keep Loehle out of the literature) even while Juckes et al 2007 is accepted.
25 JEG
I think we are dodging the issue re “peer review” somewhat. Both you and Dr Curry touch on the problem, without fully addressing it. As both an author of quite a few papers and a ‘peer reviewer’, I would like to add my comments.
The world of scientists in a particular specialized field is a small one, and the members are generally quite well known to each other. As JEG points out, reviewers are human, and it is difficult for them to give a colleague who is a buddy (e.g., on the Team) a tough review. Equally, if a submitted MS disagrees with, criticizes, or attacks the work of a friend, then a tough review is more likely.
These are the facts, and the peer-review “cachet” is not worth too much in a highly contentious field like GW.
Jeg, followed your link and found it informative. One line in particular;
“Let me be perfectly clear. I am not here to defend previous work : the sole focus of this review is Loehle’s article.”
As a “newcormer (sic) to this game”, you would be well advised to look up previous work- on which so much has been written, scientific reputations hang and which is still a poster-child for the IPPC (and a Nobel Prize winner). Contrary to what many would like, this issue has not “moved on”. Only by reviewing what has gone on with these and related papers will you be in a position to focus on Loehle’s latest work.
Please don’t digress onto generalized discussion of the pros and cons of academic peer review.
Let’s consider specific questions and try to draw conclusions from that. MBH99 didn’t report a verification r2 statistic, nor did Moberg nor did Juckes. I agree that Loehle should present one, but I’ve also requested this from Mann without any success. While Climate Audit may advocate this standard, by what right does Emile-Geay call one study (Loehle) that omits this statistic “pseudo-science” without leveling the same charge against the other studies?
JEG, welcome back. More criticism and less vitriol is a wonderful thing. Too bad we got off on the wrong foot. I hope you will hang around and contribute to working through the issues raised with Dr. Loehle’s paper. As I understand it, terms like “fraud” and “pseudo-science” generally get clipped around here so let’s try to leave that out the discussion. As you’ve stated above “no scientist has the time to deal with a zillion trolls”, but that is exactly what Dr. Loehle has willingly submitted his paper to with the hope that it can be improved. I think he is at least owed a some respect for that, and we should all restrict our comments to valid criticisms and suggestions for improvement. I hope you will agree.
RE 16. Susann.
The issue is this. from the policy standpoint you might base your actions on “consensus” or rather
you might argue that the “consensus” of scietists is:
” XYZ is true” , therefore policy PDQ is
required. When the “consensus” employs one set of rules with regards to canonical works and
another set of rules with regards to non canonical works, that raises issues about the logical coherence
of the “consensus”
I don’t think Dr. JEG has a “double standard.” I think that term far too strong. When the “consensus”
of scientists look at data that confirms their theory, the tendency is to “forget”, “ignore”, “trivialize”
particular problems. This happens in all normal science. When they look at data that tends to disconfirm
their theory, there is an opposite tendency. In short, science for the most part is a very conservative discipline.
This is not a comment about Dr. JEG. it is an observation about the structure and evolution of normal science.
I suspect JEG was not aware that his valid criticisms apply to that older work. He was probably not aware of the potential of being accused of applying a double-standard. (Not everyone knows the HS story like CA readers do.) JEG has indicated that he stands by his words, and what that means is good news for climate science: were he to review another MBH paper he would probably apply the standard equally. Good enough.
i.e. Let ALL proxy papers report their full slate of verification statistics. (Just to tie #41 back to the thread title. And also to give Susann what she wants – a solid basis for comparing these studies.)
#13
Bender says:
This issue, dating uncertainties – x-direction uncertainties, is one Dr. Loehle implicates as a problem with the other reconstructions. So, why is this recon given forgiveness for this issue?
#36
Pat Keating says
How is this different form E&E?
More directly, research indicates that people generally review a buddies paper harder than one by someone who is not a friend. So, your argument is not supported by data.
40, and it kinda has to be that way, or we’d be spending all of our time explaining for the 3,546th time why somebody’s perpetual motion machine won’t work. There’s clearly a problem with letting anything with legs walk in the door.
Forgiveness? You aren’t reading the comments. He’s recognized the potential problem that has been pointed out numerous times and is working on another paper to address it. i.e. He’s moving on. That’s science, for you: always progressing.
44 stan
In respect of Steve’s express wishes, my answer will be posted on Unthreaded #25.
RE 41. Seconded.
One thing that might be nice for lurkers is a simple explaination of verification r2.
or pointers to basic explanations.
the other day for grins I downloaded tasmainian tree ring data for the past 3600 years.
I looked at the temp reconstruction and said ” now what?”
Steve M says :
Point taken. Here’s a cheeky answer : i was in school at the time. I defended my PhD a year ago on some rather orthogonal topic, which has arguably no bearing on the issue at hand. I started a postdoctoral fellowship in Dr Cobb’s lab right after this, so my knowledge of the literature is about as recent – though i remember reading the Moberg paper in 2005 and feeling distinctly unconvinced by it : i had issues with the relative weight given to low vs high-frequency proxies before incorporation into the wavelet coefficients, and – call me partial – i was convinced by Team’s answer to this. (Mann et al, JClim 2005,p 4103, 4104). As for Team’s work, i again refer you to my disclaimer (part 3). Since i have never endorsed any of the “Team”‘s work, it is simply inappropriate to ask me to justify it. I have my issues with some of the older one, but it think it would be fruitful for CA-dwellers to get up to date with their most recent work. Of course it is not perfect – nothing ever is – but if i were to apply the criteria i used in my review to their work, they would pass the test. The point is that while no author can be expected to bear the world on his/her shoulders, there are essential questions that must be addressed (either with revisions or rebuttal) in any work. In my book (which appeals to no other authority), Mann’s recent work does, Loehle paper does not (in its current form).
Now, I know i have high standards (i almost defected to the Dark Side of financial maths because i was tired of reading overblown pieces of junk in some of our own peer-reviewd journals), and i do my best to apply them to my own work. The day will some come when i publish my own multiproxy NINO3 reconstruction (with Drs Mann and Cobb, sorry), and i had long decided to simultaneously release all my code, and as much data as i can. CA, i promise, will be the first to know. The problem here – which am sure will not convince any of you – is that while some paleoclimatologists are willing to openly share their data, others only do so for colleagues, making it hard to keep cordial working relationships if you don’t maintain confidentiality. I wholeheartedly agree that this practice should cease, and all should be accessible to all – but there is some inertia in the system . I think CA’s efforts in this regard have already been crowned with much success, so please keep pushing. However, please understand that we are not always in an easy position.
More generally, i think a lot of CA discussions would benefit a great deal from a deeper understanding of some basic physical concepts (like “teleconnections”), and the structure of our data. What i read on these pages often underestimates the difficulty to obtain paleoclimate data, where it can take years of work to retrieve and analyze a sediment core, for instance. The result is that many authors are emotionally attached to their baby and have a hard time letting go of it. It is obviously easier for me to criticize, as i sit behind a computer and their precious data is all but a table to me. Clearly many of them (no names…) would be wise to let go a little, but i wish CA-dwellers had a deeper understanding of how this data comes about instead of rudely demanding it like it’s their birthright. Perhaps, in the end, everyone will understand each other a little better.
In this regard, i give kudos to SteveM for personally going to “the ass end of nowhere” to retieve tree-ring data, instead of being an armchair skeptic. That is the way to do it, and i hope i will shake many dendroclimatologists out of their seat to get up to par. We as a community have more than one gauntlet (sp ?) to pick up from CA.
Finally, I’ve read somewhere on CA an equally cheeky comment about my sin of being young, something about “experts being fresh-baked daily” in climate science (can’t find the post now, sorry). Never have i pretended to be an expert – i despise the term myself, since anyone can always be an “expert” at anything, providing they choose a narrow-enough field. Since CA prides itself in valuing’s anyone’s opinion, i simply think mine is as good as anyone else’s – but no better. You are free to disagree and i enjoy it when you do so constructively. I hope this sets me apart from Team and the Royal Climate mentality, and that you will respect it.
Back to work…
Steve: A quick point. I did not go to the “ass end of nowhere” to retrieve tree ring samples. We met at a Starbucks in Colorado Springs and then proceeded in a comfortable 4-wheeler to a relevant bristlecone site, sampled trees and were able to return for dinner. It took a couple of trips to locate the Graybill samples only because he failed to make even a simple map. One of the points was that Mann’s excuses for not updating the tree ring proxies – it was too difficult and too expensive – were made without any basis and could be readily disproven. Of course, some ice core proxies take somewhat more effort, but it’s not an impossible effort. Such difficulty does place a greater onus on authors such as Lonnie Thompson to make a complete and comprehensive archive of all his sample information as it is not practical, for example, for me to go to Tibet and take my own ice cores because Lonnie Thompson won’t archive his results. It is dispiriting that climate scientists have acquiesced in such obstruction and obfuscation.
#52 Nice comment.
Although youth is not the sin. The sin is arrogance.
Is it IP? Is there a reason to restrict access? If so, that’s between the scientist and the funding agency, but it severely undercuts the credibility of the paper if the raw data can’t be accessed, and the analysis can’t be reproduced. Or at least it should. No one has the right to advocate policy based on proprietary data.
Limiting access is fine for commercial ventures, it has no place in science, particularly where there’s a rather huge policy question being impacted.
Julien, I appreciate the cheerful tone of these comments. You say:
Well, I think that I make a pretty good effort to be up to date. I’ve commented here in near real time on Moberg et al 2005, Osborn and Briffa 2006, Hegerl et al 2006a, Hegerl et al 2006b, Juckes et al 2007 and IPCC AR4 (which still includes MBH, Jones et al 1998, Mann and Jones 2003, Briffa 2000, Briffa 2001, Esper 2002.)
In some cases, it has taken several years to get data and adequate methodological description from authors, delaying my ability to comment, but I do the best I can. In this respect, Loehle has been very cooperative relative to (say) Esper.
As to your statement that this most recent work would pass your test, I can assure categorically that Juckes et al 2007 and Hegerl et al 2006 – to pick two very recent studies – do not. Perhaps you can give ma citation to a paper that meets your test.
Also do you agree that Juckes et al 2007 and Hegerl et al 2006 are “recent” enough that standards applicable to their acceptance are relevant standards for Loehle?
Re #54
I believe I explained myself adequately in #48. Shall we cut our losses, call it even, and just get back to work?
Steve: yes, please: enough of this idle banter.
JEG said
As one who processed and counted microfossils in hundreds of such samples many years ago, I’ll second this statement. However, the high cost is all the more reason to strive for maximum confidence in the method, high standards of data acquisition, and free sharing of the information.
BTW Mann’s most recent article Mann et al 2007 does not result in a reconstruction that passes a verification r2 test.
His present argument – like Wahl and Ammann – is only that a verification r2 failure shouldn’t count against them.
If Emile-Geay agrees with Mann that verification r2 and CE failures don’t “matter”, then why is he demanding this information from Loehle.
Personally I think that this information is relevant and, until paleoclimatologists can produce a model that passes a verification r2 test, then it is hard to find an objective basis for preferring one reconstruction to another.
So you don’t endorse the team’s work, but you will sign on to do work with them. Interesting.
I would only feel comfortable with co-authors whose work I endorse. But maybe that’s just me.
OT, but something I completely understand (as opposed to the above. Having their students end up in the finanical world has forced many a Columbia science or engineering prof’s head to shake.
I think you need to tread carefully when trying to analyse Loehle 2007 in isolation.
There is an issue with pretty much all multi-proxy reconstructions. They simply aren’t robust; whilst it is always possible to “demonstrate” robustness by giving anecdotal samples in a paper, it only takes one illustration of the lack of robustness to falsify this. Subsequent demonstrations of robustness cannot then reinstate the claim.
It is well known, thanks to Steve’s work, that the multi-proxy reconstructions are simply not robust to data selection (most rely on a small number of specific proxies; remove these and the shapes change substantially), and we also know that the reconstructions are not robust to methodology (eg. Steve’s work again, plus Burger and Cubasch 2005).
The problem is, that the scientific literature has been drenched with reconstructions that show cool medieval – warm modern relationship; but we know that it is trivial to set up the reverse, using the same standards as other reconstructions, yet none show this. This is a significant bias in the literature at present, and the reason that the NAS panel demoted the modern / medieval claim from “likely” to “plausible”.
Well, it is entirely consistent with the NAS panel conclusion that a warm medieval / modern relationship should also be “plausible”. That is why I think this paper deserves to be published (albeit with some of the comments here addressed). It may suffer from similar flaws to other studies in the field, but the fact that it gives a different answer shows a different kind of “plausible” and helps to underline the limits of the current “state of the art” of paleoclimate reconstructions. It falsifies the idea that multiple poor quality reconstructions can define a cool MWP. And kudos to Dr Loehle for not trying to overstretch his claim beyond “plausible”.
BTW I assume it would not be possible to compute an “RE” statistic for Loehle because there is no calibration period; r^2 is where it is at (and where Mann’s work should be at also). If there is insufficient content in the proxies to give a meaningful correlation coefficient, just what exactly is the regression step working on?
All just MHO as ever.
Wait a sec here. You’re criticizing me for pointing to someone’s rookie error in an over-the-top review, and you’re criticizing me for pointing out that “pseudo-science” is a serious epithet, another rookie error?
1. If I hadn’t submitted my review first, JEG would look like a hero to the warmer camp. My review was as critical, but professional.
2. Speaking of credibility, what about Loehle’s credibility? Unjustified claims of “pseudo-science” ought to be retracted. You gonna let a rookie get away with that kind of diatribe?
3. If my credibility hinges so delicately on whether I decide to get playful with a cheeky rookie, then that says something about your own judgment.
Let’s have a little perspective. I would like JEG to hang around, but he should play by all the rules the rest of us do. Good day.
“I sent my files to UC for double checking, but here are the preliminary results:
RMSE=0.067, so that gives Mannian CIs as 2*sigma=0.13! BTW, R2=0.73 and the series are remarkably similar using the Mannian terminology.”
Was your methodology the standard practice? Simply curious.
Steve M: you may wish to remove my #58 as it no longer refers to anything. Apologies for the indulgent behavior.
#48
I found this a quite useful explaination.
Steve, any chance to simply remove the text body (or replace is with [snip]) but otherwise leave the posts in place when you delete? So many folks reference previous posts with number and this gets a tad confusing when deletions occur.
OK, let’s tackle Steve’s blog question here, on “the dissonance between Emile-Geays demand for a verification r2 statistic from Loehle as compared to past contortions by Mann (and Ammann) in trying to cover up the MBH verification r2 failure.”
In my original review, i was asking for some amount of cross-validation. The idea is, as most people would agree, that one can only trust a reconstruction in 1100 A.D. if it has proven apt (or “skillful”) to reproduce a salient traits of the XXth century instrumental temperature record.
Craig Loehle did not do this because, he argues, “he is not using a model”. He is (see my blog post and bender’s #170 on the original thread).
I still stand by my claim that Craig’s failure of even touching on the issue is either woefully naive (in which case it does not deserve publication) or downright unethical (in which case it does not deserve publication). That’s why i called it “pseudo-scientific” : something that has the deceiving appearance of science but lacks rigor in the crucial fulcrums of its reasoning. I am sorry if the term “pseudo-scientific” upsets readers as much as reading the article’s lack of rigor upset me. Nonetheless, I will stand by it until someone demonstrates that this issue has been given serious consideration.
Now, there are various ways of conducting this verification : CE, RE, R-squared statistics are only several statistics amongst many. The “dendros”, as they are affectionately called here, have been working on this for decades( see Fritts, 76 and references in Esper et al 2002, for instance). Apparently their methods do not satisfy the wise among you : so what should be done ? This is a topic i am actively researching at the moment, and on which i would LOVE to get input from the wise statisticians who swim these waters. There are industry standards (which Loehle did not observe – and the fact that Mike Mann didn’t either is no excuse), but that doesn’t mean they cannot be improved.
Now, why ask Craig Loehle to provide R2 and not Mike Mann ? That was indeed cheeky. I do believe metrics like RE and CE are useful (and were an example of what should have been provided to convince the reader), while an annual R-squared would be ill-suited to evaluate the skill of reconstruction in its ability to track low-frequency (say, period >20 years) changes, which are arguably the ones of interest as far as the response to external climate forcings is concerned. “Team” did explain the pitfalls of using the R-squared statistics here and here. I am interested to hear what is wrong about their rebuttal. (There may be a CA post on this – please point me to it).
On the other hand, if one is making claims about the skill of the reconstruction to track yearly changes (as MBH99 implied when they say “1998 is the warmest year in at least a millennium”, then R-squared should be reported and as said before, i am not here to explain Michael Mann’s behavior . Since i am de facto on the hotseat for him, all i can say is that i would have done things differently – but that is cheap to say since i was not in his shoes.
So i was asking for CE and RE, mainly, but could not help but asking why R-squared had not been demanded by the R-square-fanatic audience. It was misplaced, as Steve M himself proved to us today (cf “a demand with which I agree”). So my point really is : why is there no cross-validation whatsoever here ? (which bender had historically asked earlier, apparently, though i was too busy writing to read his comment in time : mea culpa).
Do i have double-standards ? Not in my own work, which is what matters. In fact, it hass been said by much more eloquent a writer :
So now that Steve M and I agree somewhat, let’s focus on the real question : what is the most meaningful cross-validation approach to use here ?
Now, re #5 :
Thank you. That was a very interesting point. What, then, should be done to estimate how accurate a multiproxy reconstruction is back in time ? Is it even possible to devise meaningful confidence intervals ? These are research question of prime importance and i am curious to hear your advice on it, Jean S.
We are on the brink of another 500 comment thread here, as Steve M, bender, and JEG (with probably an occasional concise comment from UC or Jean S, go back and forth and in and out of Esper, Juckes, et. al, ad nauseum, etc. JEG wrote that Steve M had a “decent grasp of mathematics” or something to that effect, and he obviously thinks very highly of himself. Somehow, though, I don’t sense that this will lead to any sort of closure or “consensus”.
What is desperately needed in this re-construction morass, is to recruit the active involvement of statisticians with solid credentials to co-author the papers under consideration. These papers are, after all, statistical exercises, and really don’t have anything to do with the topics that are the provenance of climate scientists. In almost any other scientific endeavor which depended so heavily on a specific discipline, this sort of collaboration would have been the first order of business.
Why is this so hard?
Re #64, Tom C
Because climatology is a political endeavour.
Because:
-the statisticians are being told that “the science is settled”
-there is not much funding for re-visiting “old” questions (the focus is now on “adaptation”, as opposed to causal “attribution”)
-those asking honest questions risk being ostracized as earth-haters
-there is bascially little reward for the amount of risk
You will need to be patient. If JEG is serious about robust estimation of confidence intervals – and I believe he is – there is hope yet for multiproxy science.
Better scientific transparency is vital to gaining clarity concerning so many of the issues we’re bickering about. I am reminded of an ongoing intellectual property lawsuit between SCO and IBM.
SCO and IBM worked together in the late 90’s to enhance the aging unix operating system, which is owned by SCO. The project fell apart when IBM backed out and decided to concentrate its efforts on improving the open source operating system called linux. SCO then sued IBM and other companies supporting linux, claiming IBM stole code from SCO’s unix os and incorporated it into linux. SCO, waiving arms wildy, publicly stated “Linux contains Millions of lines of unix code”. SCO hoped, through the tried and true threat of lawsuits, that IBM and others would settle out of court, and this threat would scare the bejesus out of business and software users who were thinking about switching from unix to linux. The one thing SCO didn’t factor in to their plan was the ability of community experts, via the web, to quickly debunk the threats.
To prove just how much unix code was in linux, SCO released numerous examples of infringing code to show just how corrupt linux was. As it turned out though, all the examples provided had come from older code, written by AT&T, that not only predated SCO’s unix by several years, but was also legally in the public domain. Linux, as required under the open source GPL license, cannot hide or conceal the underlying code; all must be easily accessed by the public. This made it easy for geeks and bloggers, before IBM even had an opportunity to file the first bit of legal paperwork, to quickly trace the origin of the examples provided by SCO and show the claims to be false. Four years later, not only has SCO NOT sued IBM, or any other company over claimed code infringement as it originally had threatened, but SCO, as it turns out, does not even own the rights to unix at all (it’s complicated). The multiple lawsuits brought by SCO are not going the way they wanted, and the company is now in the process of filing chapter 11.
How is this relevant to the issue at hand? The art of lawyering is often more about the presentation of the evidence vs the actual “correctness” of the facts (how many times have we scratched our heads over a verdict we absolutely disagree with – O.J. – I’m looking your way) Software issues presented in the court system mirrors, to a great extent, the way science is understood within the general public. Science is HARD to present to both the general public and a jury (they are one and the same after all). Juries are not, by the very nature of public diversity, experts in any particular field, making it easy to get around scientific uncertainty by appealing to their emotions. I worry that too many on both sides in this climate science debate fall back on the same methodology to sway the public in their favor.
It’s impossible to say as IBM more than likely would have found the same evidence against SCO’s assertions, but if the geeks and bloggers hadn’t so quickly debunked the initial SCO claims, it’s possible IBM and other companies may have settled and SCO would have, as a result, effectively killed linux. Instead, the open source nature of linux prove to be a safeguard against false claims, and because any errors or flaws found in the code can be fixed quickly, open source gives linux operating systems an advantage over proprietary systems such as Windows XP or Vista (it has often taken Microsoft months to patch security flaws) I think an open source vs proprietary philosophy of data, location, methodology, etc. would go a long way to enhance the reliability and robustness of climate science, and all sciences.
Hope this made some sense.
#52 “i wish CA-dwellers had a deeper understanding of how this data comes about instead of rudely demanding it like its their birthright.”
JEG, this research is funded by taxpayers like me under science grants. I don’t expect it as a birthright, I DEMAND IT as a taxpayer, especially when it is being used to dun me for an ever greater portion of my income to pay for mitigation based on unproven theories. I want the science verified by duplication. That is difficult to do when the data and methods are not made available as they are supposed to be under the rules of Federal grants and the granting institutions as discussed many times here on CA.
Strictly as an observer, the current state of CS peer review and validation seems shameful at best and a non-existent, good-ol-boy, pat on the back at worse. In fact I’d go as far as saying that CS peer review is simple farce.
Lastly, I’m okay with holding Dr. Loehle to a higher standard than the Team. As more studies are added to the list and that standard is held (hopefully), I think the Team’s research will be found to have a foundation of sand and collapse while Loehle’s and others prove that AGW is pseudo-science.
#53 Bender, arrogance defines climate science.
67, not exactly. SCO doesn’t own unix, and never did. It was developed by Bell labs, and eventually found its way into the public domain. The contested code is part of SCO unix, which SCO claims was proprietary, and somehow was illegally introduced into the open-source Linux project, which IBM uses. I won’t comment further, because it’s a complicated case, but most observers believe that SCO doesn’t have a case, and they’re just fishing for a settlement.
Loehle has suggested that his reconstruction can be improved and is amenable to replying to rather strong criticism of it. Mann, on the other hand, appears to me to be of another mind in matters such as this. Mann calculated RE, CE and R^2 but chose not to report R^2, but you should not confuse and applaud the mere fact of reporting a statistic with using the statistic appropriately that would not avoid the charge of suspending judgment.
By the way, JEG, you need not answer a zillion trolls for those of us, in at least the average perception range, to quickly zero in on where you are coming from. In blog exchanges, personal declarations, that one is above the fray, without any double standards and approaching the general subject of temperature reconstructions with a cold and calculating scientific mind, can be belied by the tone with which the declarer posts.
RE #63 (JEG):
This is the fundamental question. Without an answer, no reconstruction is worth anything. So I would suggest focusing the discussion on this one point.
You seem to be of the opinion that recent work by Mann – eg here – is much better in this regard. Insofar as the question of extapolated confidence intervals is concerned, how is it “better”? Whatever technique they use to derive extrapolated confidence intervals can presumably be applied to Loehle’s reconstruction, so your answer will provide a good basis for comparison.
Tom C.
Furthermore it is exactly this impatience that you display that got us here in the first place. None of the multiproxy studies should have been published (not MBH9x, not Moberg, not Juckes, none!) without serious disclaimers as to how unskillful the reconstructions were. All of it is more-or-less “pseudo-science” if you were to apply JEG’s criterion across the board.
These statistical challenges are not at all trivial! You may as well complain “why can’t the programmers get the GCMs right?”, “why can’t we measure earth’s surface temperature everywhere?”. These things are easy to ask for, hard to deliver on.
#72 bender
I understand this. What I am getting at is that the Team can endlessly disparage Steve M. because, despite that fact that his mathematics are superior, since he does not have formal credentials. They would not risk the same rhetorical assault on a Wegman, for example.
The same principle cuts the other way. Maybe the Team is right. But they will never prove it until their results are validated by the specialists.
Proxy selection is tied to the physical sciences of chemistry, physics, botany, whateer. But reconstructions are a purely statistical exercise that takes place once the proxies have been selected. A climate scientist is no more likely to do a good reconstruction than a civil engineer.
I can’t understand why no statistician appears wiling to step forward and make a name for his or herself by tacking this stuff. Is the political price to pay that heavy?
A more immediate question is to ask what the outcome was of the first workshop on statistical climatology.
JEG? Anyone?
Re 64 This has been pointed out, but the situation remains:
“Wegman found that Mann made a basic error that “may be easily overlooked by someone not trained in statistical methodology. We note that there is no evidence that Dr. Mann or any of the other authors in paleoclimate studies have had significant interactions with mainstream statisticians.” Instead, this small group of climate scientists were working on their own, largely in isolation, and without the academic scrutiny needed to ferret out false assumptions.
Worse, the problem also applied more generally, to the broader climate-change and meteorological community, which also relied on statistical techniques in their studies. “[I]f statistical methods are being used, then statisticians ought to be funded partners engaged in the research to insure as best we possibly can that the best quality science is being done,” Wegman recommended, noting that “there are a host of fundamental statistical questions that beg answers in understanding climate dynamics.”
In other words, Wegman believes that much of the climate science that has been done should be taken with a grain of salt — although the studies may have been peer reviewed, the reviewers were often unqualified in statistics. Past studies, he believes, should be reassessed by competent statisticians and in future, the climate science world should do better at incorporating statistical know-how.
One place to start is with the American Meteorological Society, which has a committee on probability and statistics. “I believe it is amazing for a committee whose focus is on statistics and probability that of the nine members only two are also members of the American Statistical Association, the premier statistical association in the United States, and one of those is a recent PhD with an assistant-professor appointment in a medical school.” As an example of the statistical barrenness of the climate-change world, Wegman cited the American Meteorological Association’s 2006 Conference on Probability and Statistics in the Atmospheric Sciences, where only eight presenters out of 62 were members of the American Statistical Association.
http://www.canada.com/nationalpost/story.html?id=22003a0d-37cc-4399-8bcc-39cd20bed2f6&k=0
#64
I wonder. I have been actively seeking collaboration with statisticians on the topics, but got frustrated.
There is a quote by Peter Niiler , (i think) that goes like “Scientists want to do science, and modelers want to do models”. For the topic at hand, wee could say “Climatologists want to do climate, and statisticians want to do statistics”.
After 2 workshops in statistical climatology, i got a distinct impression that the sort of questions we, climate blokes, are interested in, are not exciting on a statistical point of view. And conversely, most of the talks i heard from statisticians (with titles as lovely as : “asymptotic distribution of eigenvalues in random matrices”) seemed very removed from the practical issues we face.
Thanks to CA, the World knows what is wrong with climate scientists. Perhaps there’s also something wrong with statisticians ? I do believe a lot of the disputes here are a question of language – ah, Wittgenstein !
That could explain some of the frustration between the two communities. I just re-read the summary of the Wegman report and i had to laugh : they really do not understand *anything* about the proxies they were looking at. On the other hand, i understand when statisticians get upset at the improper use of statistical terms that many of us (including me) may be guilty of.
Since you guys are so darn constructive all the time, why can’t we broker a truce ? That would be progress. Someone earlier asked for good references in statistics, and it would a very useful contribution to assemble an “essential reading” list.
“Why is this so hard?”
Practical problems do not necessarily interest theoretical statisticians. Try to find a coherent explanation wrt to when to choose what wavelet, how to choose the correct block size for time series resampling, etc. You’ll find many bits of information that require a graduate level understanding of statistics. Most people do not have this. They know their own toolbox and nothing more. And while you’re wanting to understand an implementation issue of a novel statistician technique, your author has moved on. There are some exceptions, financial types come to mind, but the problem remains that the innovators, the people pushing techniques that might be helpful here, are not interested in coming up with, say, useable Matlab code for an applied setting. Unless you have money.
#80
Agreed. Statisticians face the same publish-or-perish pressure that the climatologists and dendros face. What is required for tenure is not correct papers that move policy forward, but novel papers that move the science forward. So you have each specialized domain forever specializing itself into irrelevance, with little chance for cross-pollination. This problem is recognized, and some universities are viewing it as an opportunity for a renewal in “interdisciplinary” research. But it’s a slow process. There is still far more reward for specializing than for generalizing.
Getting the pure statisticians interested in applications will continue to be a challenge. What would help would be a fund directed at that goal.
Yet Steve has critiqued several papers, only to be rebuffed – his criticism was valid, but ignored by the “peer reviewers”. Just as JEG’s critique was valid, so was Steve’s
Try changing a few words in that first sentance to see my point:
JEG:
Estimates are that hundreds of trillions of dollars are going to be spent based on policy recommendations that derive from this data. It borders on the criminal for some of these scientists to be withholding that data. This is professional malfeasance of a very high order.
I understand the feeling of ownership of intellectual property. (I’ve had articles published.) You suggest they are being protective of it, as they would their “baby.” But there are other possibilities that present themsevles, i.e. that the data is flawed or incomplete and they don’t want it to be exposed publicly that way. If that’s the truth of the matter, then their behavior is unconscionable, given what is at stake.
If the data is good, it will stand on it own and they will be recognized for their contribution, which, if the often dire predictions based on their work prove true, will be profoundly valuable in an historical sense. They will be recognized for having contributed to warning humanity of impending catastrophe. If they believe in their work, why not gamble on that?
Honestly, do these problems require the work of the top theoretical statisticians? I can understand that the top academic statisticians might not be interested in doing the seemingly mundane work of applying statistical techniques to paleoclimatological data, however, the world has a lot of competent statisticians that do applied work every single day of the year, and in multiple fields of work. Why not look for collaboration with one of those?
JEG, the problem is that you’re thinking of the wrong kind of statisticians. IMHO the people that are most likely to be helpful with the sort of statistical problems relevant to paleoclimate are econometricians and business statisticians, not in their capacity of being interested in economics, but in their capacity as people familiar with spurious regression and autocorrelation.
I urged NAS to include someone with this background on their panel, but they chose not to.
Instead of reviling McKitrick and me as “amateurs”, climate scientists might have reflected a little on the fact that we’d obviously noticed problems and that there might have been something in our backgrounds that was enabling us to see things that people with more conventional backgrounds were missing.
I personally think that there are many statistical problems that I’ve encountered that could be far more interesting to a stats grad student than whatever they are working on.
#63 JEG. All the Hockey Team validate their proxies by determining their fit to the data in the calibration period, in this case part of the modern temperature record.
The (simplified) way this is done is to add up the differences between the proxy data and the calibration data. So for any year the “thermometer” temperature is subtracted from the proxy estimate of temperature. If the differences are not too great the proxies are said to co relate with the actual temperature. Naturally as all the Proxies show nearly the real temp in the calibration period, they are judged on that basis only to be good proxies.
The average of the proxies is then extended back in time, to show what is alleged to be a temperature in the past.
However, in all cases of the Hockey Teams reconstructions, the sum of the differences of the individual proxies estimates of temperature, and the average proxy estimate of temperature outside the calibration period is so great, that if the non calibration period proxies had been tested for co relation with their average, in the same way as the proxies were tested for co relation with actual temperature in the instrumental period, they would mostly have failed that test.
This can simply be verified by looking at any of the graphs of each proxy side by side, something that SteveM has helpfully provided many times. When you see all the graphs side by side, it is readily apparent that while they may mimick each other in the instrumental temperature period, they mainly go wildy out of alignment in the period they are being used for the reconstruction of past temperatures, and therefore they cannot all be said to be good proxies for temperature.
If JEG would therefor show, for any of the Hockey Team Papers, the result of any standard statistical test which verifies the use of their proxies for temperature by reference to the difference between them individually, and their average outside the calibration period, and which by the same standard verifies the use of the proxies by reference to the difference between them and the actual temperature within the calibration period.
76, JEG:
Excellent. That is just what is needed to move forward on all this.
JEG writes:
For the record here is my offer of Dec 12, 2005 to Caspar Ammann – preceding both the NAS panel announcement.
In early 2006, after Ammann failed to even acknowledge this offer, I reported on thr proposal as follows:
All this is acknowledged. Mr. McIntyre was rebuffed and ignored and treated like a troll and outsider. He has had more of an impact on climate science that probably anyone else since its inception, given that there has been a congressional inquiry into MBH and a review by the NAS. I would expect, after the revelations about the problems with MBH reconstructions, that people at CA would expect — demand — that Loehle be held to the proper standard, rather than the same inadequate standard that MBH appears to have been held.
If you are truly interested in moving the science forward so we can feel that the science is valid, you will expect all paleoclimate work to be held to the same higher standard. Otherwise, it just sounds like sour grapes.
In all this, kudos to Loehle how has sounded the most professional by welcoming criticism and looking eager to move forward to improve his work.
#86. I don’t believe that I’ve said anything that can give support for the view that I’m being any less critical of Loehle than similar papers with different results.
As someone whose background is not academic, I have trouble understanding what the “proper standards” are for a paper like this. And please bear with the question. If Loehle meets the same standards as work such as Hegerl et al 2006 or Juckes et al 2007, is that sufficient for it to be published? As a reviewer, that would seem to me to be a relevant objective standard for determining whether a paper should be published – not whether I agreed with how he did it. The latter seems far too subjective – those are just my opinions. It seems to me that academic reviews are all too often very subjective – people in the system are used to it, but, coming at it like an anthropologist from outside, while many aspects work decently, it’s still merely one form of screening.
What if Loehle’s paper was vulnerable to the same criticisms from me as the other (recently published) papers? If those other papers met objective standards, then what right do I – in my capacity as a reviewer – have to prevent publication if the paper meets the same standards. If I hold otherwise, isn’t the reviewer then merely inserting his own POV into the matter and no longer functioning objectively – something that seems all too common to me.
I think that it’s quite possible that a reviewer could disagree strongly with a paper and even disagree with aspects of the presentation (and in his capacity as an author express these disagreements), while at the same time, in his capacity as a reviewer, agree to the publication of the article.
I ask these questions because I don’t have a long experience in academic reviewing and I honestly don’t know the answers. The few reviews that I’ve done look completely different than anyone else’s reviews. They are far more detailed.
Personally I think that reviewers should spend time ensuring that the data and code are properly organized so that the results are replicable immediately. We’ve had to spend time on this in connection with Loehle and I think that that’s too bad. However we will likely end up with a better documented study than his rivals.
Susann, 86 – I think Bender’s already addressed that. I think there’s general agreement that better quality is better. The question that’s being wrestled with here is how good can the temperature data be that’s inferred from trees and mud. I think we’d all be delighted if there was a way to read temperature in the past from mud, and be 90+% certain that it’s accurate to +/- 0.1 C, but I’m afraid that’s not in the cards, not now or in the future.
Where we’re at now is trying to get a handle on what the most accurate reconstructions are, and what’s a good estimate of their likely error. And that goes for the work of Mann as well as Loehle. This is why we keep getting sidetracked on Mann. We’ll deal with Loehle, but in the process, we keep hearing valid criticisms, which I believe that Dr. Loehle will deal with, which were never heard from the same quarters when Mann was doing similar work. Yes, it’s a distraction, and yes, I believe that the Loehle paper will eventually be corrected, but it’s not irrelevant from a policy standpoint, because the MBH work hasn’t achieved the closure that should be by now.
JEG #76, November 19th, 2007 at 2:17 pm says:
Seems like a derisive comment with little scientific content.
I don’t recall claiming anything to that effect.
I don’t really understand. That lack of valid criticism was a problem, no? Are you suggesting that since people were lax with Mann, they should be equally lax with Loehle? Or do you expect people to distance themselves from Mann and others as some kind of litmus test? Must those who criticise Loehle preface all criticism with similar references to failings in Mann’s work? I don’t think that’s a wise move, as satisfying as it might be.
Steve McIntyre says :
Then why do we keep hearing that mine are not valid arguments because they were not used against Team at a time where i wasn’t even around ? I agree with Larry (#89) that constantly whining about how Mann did not play by your rules is weakening the current debate.
Several well-posed questions have been asked to Craig Loehle, by many CAers and yours truly. It is laudable that he so willingly submits to this type of snake-pit-review process – which would scare away many an author, regardless of the field. So i’ll respect this good will, wait for a few weeks and see what he has to say. If he proves to answer criticisms adequately i’ll do the right thing and retract my statement that this is “pseudo-science”. That said, i reserve the right to examine his work further and submit criticisms if warranted – which i will most likely do on my blog.
Meanwhile, if anyone has issues with Michael Mann’s work (and i’m not looking at anyone in particular), they should :
a) stop asking me to defend it.
b) undertake a point-by-point criticism of his latest JGR article.
c) if they feel generous, come up with constructive ways to improve that, and any other paleoclimate reconstruction.
Have a happy holiday !
Julien Emile-Geay
Re #92:**Must those who criticise Loehle preface all criticism with similar references to failings in Manns work? I dont think thats a wise move, as satisfying as it might be.**
Not a wise move?? This blog is about verifying ALL work not just Dr. Loehle.
So it is up to you, JEG, and all the others to treat all the papers equally. We have too many who concentrate on one paper, especially if it questions the Hockey Team. The suggestion is not to be lax, but to be thorough with all, including Mann. We have not seen too many question the lack of archiving. Have you been to RC and asked about archiving?
Steve:
I believe he should have leveled the charge at the other papers.
He seems to buy into the Mann notion that R2 is meaningless for the multiproxy method. If that’s true, then Mann’s work is just junk science at it’s finest. Why?
Because according to Mann, not only did he not provide an estimate of the error, he is in principle unable to do so. Any metric that can not be assigned a meaningful uncertainty is for all practical purposes, just useless. In another word, just “junk”.
I think what Steve is saying is that claiming Loehle’s work is “pseudo scientific” while not applying the same label to the whole host of Hockey Team papers from MBH98 to Juckes et al 2007 is hypocritical. I don’t think it is applying a litmus test at all to ask people to admit and recognize that the same set of errors are present throughout the entire catalog of Hockey Team reconstructions. What I have seen in this give and take is everybody at CA saying “yes, we agree Loehle’s paper has issues and he seems committed to trying to fix them”, while JEG and Susanna seem to be saying “we don’t want to talk about how every relevant reconstruction published through peer review in the last decade has had the same set of errors…we just want to discuss how Loehle’s paper is “pseudo-science” for having those errors in it.”
Susanna and JEG, if the errors JEG has pointed out in Loehle are critical errors that cause you to deem his work to be unscientific, then will you admit publicly that the entire Hockey Team catalog of reconstructions that suffer from the same problems are unscientific and should be withdrawn from publication until they can be properly revised? If not, you are being hypocritical and wasting our time here discussing these issues. Nobody at CA is defending Loehle’s lack of statistical evaluation. We all agree it should be done and would make for a more compelling paper if it were included. But we also recognize that his paper met the same standards as papers that have been considered the centerpiece of historical climate reconstruction by the IPCC. If his paper is “pseudo-scientific”, then the entire IPCC chapter on climate reconstruction should be labelled with the same derisive tag, because it is based on papers that have the same flaw’s as Loehle’s.
96
That sums it up very nicely.
Yes, but we were discussing Dr. Loehle’s work, not Mann’s or any others. It seems entirely comprehensible to me that the subject of the thread was Dr. Loehle’s work. I’m sure Dr. Mann’s work has received ample attention on this blog. Or is criticism of Mann a requirement for standing here?
I don’t get it — Dr. Loehle’s paper was up for discussion. Not Mann’s. The focus was Dr. Loelhe’s paper. Why would JEG criticise Mann as part of his critique of Loehle?
Mann was not the issue!
Or is Mann the only issue here?
It was entirely valid for Mr. McIntyre to request the data, and it was entirely unprofessional for Dr. Mann not to provide it. In that vein, other scientists should be able to request data from CA members who publish their work, Dr. Loehle included. Must one mention that Mann was uncooperative when doing so?
First, I never said that Dr. Loehle’s work was pseudo-science. That was JEG. Please don’t put words in my mouth (or in my post, I should say).
Second, I have acknowledged that the MBH work was flawed. I’ve read Wegman, the NAS and Mr. McIntyre’s writings. I am not going to call all IPCC chapters on climate reconstructions pseudo-science since I didn’t claim Loehle’s work was pseudo-science. I still claim that a work should stand on its own merit and if it has errors or flaws, they should be pointed out in their own right. To expect that all previous flaws in all previous papers must be mentioned to pass muster is illogical and smacks of some form of petty tribalism.
JEG:
For the situation to be truly equitable and the process truly open both you and Steve Mc should be able to go to Real Climate and contribute posts that do b) and c) without fear of censorship. And Dr. Mann should come here and answer rigorous questions the way Dr. Loehle has.
That’s not going to happen because Dr. Mann is claiming a dispensation of some kind when it comes to defending his work in locales other than the close-knit climatology community.
Susann & JEG:
You are right to question peoples work. Steve Mac is also right to do so. I think what others are suggesting is that there *appears* to be plenty of “main-stream” CS people ready to disparage work that goes against the consensus, yet very few who appear ready to apply the same standard to work that supports the consensus. I don’t know why this should be, but it certainly seems to me to be an accurate description. If you can agree that this seems to be the case, I think we can then “move on”, and perhaps you will have learned something and apply a more critical eye to papers that appear to agree with your own position.
Just my $0.02.
JEG in #92:
You don’t need to defend Mann’s work, but given your criticism of Loehle and your question in #63:
it seems reasonable that you tell us whether you think the latest JGR article succeeds in constructing such meaningful confidence intervals and why.
MarkR’s comment in #83 also seems pretty relevant.
Susann
As far as I’m concerned, this is up to the discretion of the blog host. Unlike RC, this is funded out of his pocket, and a few donations in the tip jar. No grants from NGOs like at RC. So if past battles stick in his craw, I think it polite to indulge them. It’s at worst a distraction, but it doesn’t preclude doing anything right.
If you stick around a little longer, and start to pick up some of the background and history, it’ll make a little more sense. In the mean time, the comparisons to Mann may not be the topic of this thread, but they don’t preclude it, either.
When doing a multivariate regression, as JEG seems to want, one starts with proxies and temperature data and develops a model, perhaps for multiple sites at once (if one is feeling fancy). This step was already done by the authors of papers whose data I use, who converted proxies to temperature using a model and show an R2 value. Please go look at those. I could have discussed this, and probably should have, but corr with recent temperatures (as Steve M did) is not the only r2 available. Please note that this does not prove for any proxy that it is valid for past times (out of sample application) due to all the unknown things that could change.
Susann
Kudos for you with this statement. I really do hope JEG would make similar statement. It is not right to put a label of “pseudo-science” on Loehle’s work. Especially nowadays this kind of unfair name-calling spreads like wildfire around the net and as we all agree Loehle’s work is no more “pseudo-science” then any other papers in this field.
JEG only excuse seems to be that “he was not around” when older papers were published. Still, JEG should understand that many of these older papers are still referenced and held in high esteem in certain political publications. Because of this, JEG cannot just dismiss the matter by saying “I was not around when those papers where published” aka “I would have critized those papers but now it is too late”. That is hypocritical. I have also just finished my post-doc and understand that JEG might be in some kind of “high” with all the success he has got, but his present attitude is not a appropriate amongst scientist and this could cause JEG more harm in the future then he thinks.
Susan – you sum up this blog very well with posts #97 & 98. Kudos, like someone else mentioned, you are a quick learner.
If Dr.Emile-Geay feels that it is not worth his time or effort to defend Dr. Mann’s foundational work then I don’t see the benefit in asking him to do so. Isn’t his reticence completely understandable?
Julien Emile-Geay say’s that “Since I have never endorsed any of the Teams work, it is simply inappropriate to ask me to justify it”.
I find this statement difficult to reconcile with comments on his Blog regarding AIT:
Perhaps he did not know that he was endorsing Mann’s Hockey stick instead of Thompson’s Z scores when he said:
Causality is established, the “science is settled” lets move on and poke some sticks through bars.
It would seem a pity that Emile-Geay did not apply the same probing scrutiny to this review that he reserved for the work of Craig Loehle.
I am here because I respect the work that Mr. McIntyre has done and want to learn more about this community because of my interest in the politics of climate policy. I don’t personally have the credentials to do a thorough critique of either Mann or Loehle — Emile-Gaey or McIntyre for that matter – and so I haven’t even tried. I do not have the statistical competence or the climate science competence to do so.
It’s actually textbook Kuhn if you have read any of his work, specifically The Structure of Scientific Revolutions. If you haven’t, you should read it. When I first came across the work of MBH and MM and the whole HS debate, that’s what I thought of.
What I find interesting is this requirement by some (not all) that posters here “confess” the faith in order to “move on”. From what I have seen of “other” blogs, the same applies. As someone who is trying not to take sides, I see many similar proceses going on among the board members. I believe each side thinks they are acting in good faith and are dedicated to the truth, but have different positions on the issue at hand. My interest is in sorting through those positions so I can personally and professionally come to some conclusions about the issue and hopefully write good sound policy.
But you know? In the end, political will trumps even the best science and the most carefully written policy options. If it’s not on the agenda, nothing will come of the very best policy analysis. I’ve seen that happen. In the end, politicians weigh competing interests and make policy decisions based on that and their personal interests in getting re-elected. That’s where the real battles are fought — over political will. That’s the real battleground.
The extent to which followers of Mann or McIntyre can convince policy makers that they are in possession of the facts will determine what policy gets written and enacted, if any.
Susann and JEG:
I do not judge that we will be able to do a thorough analysis of Loehle’s recent work in climate reconstruction without reference to Mann’s (and the hockey team’s) papers. I would judge that Loehles reconstruction will have a number of the weaknesses that hockey teams reconstructions have, i.e. the analysis although relegated primarily to Loehles paper will be most instructive if it includes general references to problems in temperature reconstructions. Now if you and JEG want to join that analysis but have a major problem with mentioning Mann and the hockey team members and referencing their works, I would suppose some coded and generalizing language could used.
My point would be that as long as we do a proper critique and analysis of Loehles paper what should it matter to you or JEG whether the hockey team is referenced in the process. If a blog wanted to do the reverse, i.e. critique the hockey team reconstructions and at the same reference the problems of reconstructions like Loehles that show a prominent MWP I would certainly want to participate in that effort.
While JEGs hesitancy to include references/critiques of Mann and hockey team might be explained by his opportunity to publish with Mann, I am not as confident that I understand why you want a pure analysis and critique of Loehles work — as though it occurred in a reconstruction vacuum. Can we engage in hockey team references while posters are getting their ducks in order and as the analysis information is trickling in on the Loehle paper? I almost feel that your insistence on seeing some kind of Loehle purity in this process is part a test that you have devised to determine and/or demonstrate any double standards of the regular participants at the CA.
Susann
JEG made certain critiques of the paper. He was asked whether they applied to other papers. He keeps dodging the question. It’s not a point of him critiquing the details of other papers. Just answer the questions yes or no. Even if his comments are valid, the inference from much of his tone is that he is here to savage a paper he ideologically disagrees with. His answers to these questions show whether he is truly objective or just a well degreed troll.
Gore is a politican pushing a political message using science to further a political agenda. Dr. Loehle is a scientist who is presenting scientific information to the scientific (and otherwise interested) community supposedly to advance a scientific agenda. Apples and oranges.
JEG stated:
This is not correct. The 18 proxies used in Loehle is as large or larger than the number used in Jones et al 1998, Crowley and Lowery 2000, Esper et al 2002, Mann and Jones 2003, Moberg et al 2005, D’Arrigo et al 2006, Osborn and Briffa 2006, Hegerl et al 2006, Juckes et al 2007. In the medieval period, the numbers in most of the above studies are less than Loehle, sometimes significantly less: Jones et al 1998 uses only 3 proxies in the 11th century.
#46
Forgiveness in the sense that you feel this work is ready for publication with a few, but major adjustments. On top of the fine points you made, the dating errors for the time period in question are fairly significant depending on how the dating was made. Radiocarbon dating over the prior 400 yrs or so are not reliable. So, those proxies with only a few dates in the last 400 yrs have some serious issues if preformed by older methods. Here is an example of problems that can occur with such dating [go to the radiocarbon dating section]: http://www.texasbeyondhistory.net/hinojosa/evidence.html
There are other radiocarbon dating issues in the 300-1100 AD period: http://www.ksartifacts.info/3A%20time%20frames/tf01%20sample%20of%20curve.jpg Note the dating issues at the end of the MWP.
Given this, how can one robustly compare the MWP and the 20th century [last line of the abstract]. This has not been adequately addressed; only dismissed as a repeated argument and promised improvement with the next version. How will the revised version fix these errors? Will Dr. Loehle et al. use lead-210 & cesium dating or other dating methods for samples covering the last 200 yrs? How will the dating plateau of the LIA be handled?
Related – several of these proxies are best used for much longer time periods than the one of focus in this paper – 5k or more. Will these proxies be dropped because of the noise they add to the recon?
Re #107 – If that is what JEG thinks of Gore’s movie we are in deeper trouble than I thought with our academics. Someone help us.
Dr. Emile-Gaey (my apologies for my lax use of JEG and potential mis-spellings) is a well-degreed climate scientist who stopped by to provide his critique of Dr. Loehle’s paper. I admit I found the tone of his critique to be overly dismissive and ill-advised, given the forum. I do think there was substance to his critique, given Dr. Loehle’s own response and that of others who understand the issues better than I, so he was not being a troll unless you define troll as anyone who happens to disagree with your position or is impolite. The fact he has stayed around to continue the discussion is a sign he is serious about the matter. I think I even remember his expression of regret for the tone of his initial post.
The fact that he may or may not be reticient about critiquing other papers is irrelevant in my view for the critique of Dr. Loehle is valid or not, but I understand that it is a rite of passage for membership here. Showing one’s colours, so to speak.
Susann, it has been stated very clearly and repeatedly that it is a clear measure of hypocrisy or objectivity, not a “right of passage”.
I’m happy to see Mann’s work referenced when examining Dr. Loehle’s work so lead on, McDuff!
I stand by my position that it is not necessary to mention Mann’s work when doing a critique of Loehle and logically, it should not be a requirement unless the person doing the critique wants to place the current paper into context. As in, this is the literature on the issue, these were the limitations of previous studies, this is what the current paper is contributing, etc. That should be done by the author, I would think, so the rest of us can get to the meat of the matter.
What you and others seem to be proposing is some kind of litmus test that irks hell out of me, although I think I do understand it. IIRC, Mr. McIntyre’s criticism were beittled because he was not part of the proper crowd. While they say turnabout is fair play, it is rather juvenile to my way of thinking.
I’m sorry, but it seems to me that you are saying that unless posters indict Mann, they are unable to critique Loehle. Is that accurate? If so, that is illogical. It is quite possible to do both, one or the other and still be valid.
@Susan– For what it’s worth, I haven’t confessed the faith. In unthreaded, I said I think AGW is probable. It ruffled some feathers I think, but they haven’t kicked me out bitten my head off or traveled to my home town and slashed my tires yet..
On your point about Julien Emile-Geay’s criticism of Loehle’s paper: On the one hand, you are correct. Either criticsm is valid or it’s not.
That said, even at scientific conferences, it’s quiet common for people to compare results and methods of two papers. Contributions to the literature are incremental, and so people do sometimes expect that two papers may both share shortcomings, and yet both will be published because they nevertheless tell us something. In contrast, if the deficiency is horrific, neither should be published. This is sort of thought process is rather common even when there is no political dispute.
So, Emile-Geay shouldn’t exactly be blind-sided when asked how Loehle’s paper and techniques compare to Mann’s. How the two compare would be a relevant question at a scientific meeting too.
Moreover, this is a blog. Emile-Geay isn’t required to pop-in to volunteer his opinion about anything what-so-ever. And if he does pop in to have a conversation, it is natural for people to ask if his criticism of Loehel also applies to other papers.
After all, maybe the answer if no, that in Emile-Geay’s opinion the criticism doesn’t apply to Mann’s paper for thus and such reason. If he thinks that, he could say that. Or, he could say yes, the criticism does apply. Or, he can vanish. Or he can do as he likes.
But I have to ask you: if Emile-Geay doesn’t want to engage in conversation about that sort of thing, why pop into the blog? And if he pops in, criticizes what he wants to criticize, won’t engage in conversation, and won’t answer questions, the blog audience is certainly permitted to form opinions. Right? And that opinion may be that Emile-Geay’s motive isn’t entirely scientific.
Who is to say that opinion would be wrong?
Quoting Dr. Emile-Gaey
So, despite the fact the “extremely explicit attempts at estimating error bars” in MBH98 have been repeatedly demonstrated to be statistically and scientifically meaningless, rendering MBH98 guilty of consisting of Cargo Cult pseudo-science, it is is held up as a counter example to an accusation of pseudo-science. Should Loehle attempt to include errors bars? Of course and that is a valid criticism, but it is Dr. Emile-Gaey who cited MBH98 and now shies away from defending it.
Re #115 Susann: – **The fact he has stayed around to continue the discussion is a sign he is serious about the matter””
Yes, he is serious, but serious about what? If he is really serious about the subject, as opposed to ridiculing Loehle’s paper, he would objectively look at the hockey team’s papers as well. Not having looked at them or commenting shows the narrowness of his reading.
If you are wondering if Mann tells “untruths”, then just go back up to Steve’s introduction of this thread. It appears you have not read it.
Lucia, you are right — I am off base. This is a blog, a private blog at that, with a very strong and loyal community, many of whom have a very strong agenda or point of view wrt climate change and the current state of climte science – and most of all, Dr. Mann. It is not a refereed science journal listed in the ISI. Therefore, the only standards are those set by the owner. Sometimes, it has the feel of the one, and sometimes the other. I forget myself.
Susann,
You are making this discussion far more political and personal than need be. But I hope you’re having fun. Let us ask the question: what is the state of our knowledge vis à vis MWP vs CWP. You have three views of the data: (1) MWP gt CWP (Loehle et al), (2) CWP gt MWP (IPCC, Mann et al.), (3) we still don’t know because the error bars have yet to be correctly computed, but they appear to be larger larger than the difference in means. THIS is the standard which no one has achieved, but which the second side is geting ever closer to. JEG, Steve M, UC, Jean S and many others want to see robust error estimation, but so far: no go. So we argue about what the standard ought to be because it is as yet an unsolved problem. I repeat: nobody has passed the bar yet.
Again, as before: publication happens when progress is made, not when a correct answer is derived. Novelty, not correctness, is the standard. Try to understand this. This is why “flawed” studies get published. Policy-makers must learn to cope with this reality.
Re #82 and statisticians or econometricians in general.
I agree wholeheartedly.
The real difference is the kind of data you are dealing with. Put broadly, both econometricians and paleoclimatologists have to deal with crappy data. Econometricians have developed specific statistical techniques to deal with the many and varied ways in which the data can be crappy and give misleading results if the wrong statistical technique is applied. Paleoclimatologists, on the other hand, seem to have come from more a physical sciences background where the data are better (experimental even) and the required statistical techniques are simpler. This leads to problems when they approach paleoclimatological issues and seems to result in invalid inference on the basis of flawed statistical techniques a little too often for comfort.
Thus, for example, econometricians have been dealing with the problems of non-stationary variables and spurious correlation for a long time. It was first considered in the 70s, swept the profession in the 80s and was de rigeur in the 90s. Paleoclimatologists still seem to have trouble coming to terms with the implications of this in the late 00s.
And apropos of the topic of this thread, the proper validation methodology is unlikely to be RE, CE or R2 as such – it is probably a Monte Carlo study because none of the assumptions that underly the validity of the standard statistics are likely to be applicable to the data in use in these reconstructions. The joy with a Monte Carlo study is that you can bootstrap the significance levels on any statistic you like – but that doesn’t validate the statistic.
Susann said:
I have been following with some interest your recent contributions to several of these threads. You claim to have read the literature and also much of what is written on this site. If this is so, then surely you must realize how stunning your statement really is. If, as you say, you do not have the statistical or scientific background to understand much of what is posted here, how do you formulate an opinion on who is more apt to be correct? You seem to fall back on some misplaced reliance on “authority.” Having presented before two NAS panels(not related to climate science), let me tell you the last “authority” I would fall back on is a NAS panel.
re: #122
No, the NAS said they were “plausible.” They gave themselves the right to use that ambiguous word because they thought the other multiproxy reconstructions gave similar results without the bogus methods, not realizing, since they hadn’t examined them in detail, that they used the same bad BCPs and similar as produced the HS in the earlier Mann articles. You should go look at the posts here from that time frame. To summarize the problem, Mann first used a bogus method to select several BCP proxies out of a couple hundred proxies to produce a HS. The other papers simply select the BCPs and similiar directly (cherry-picking) and only had a few proxies (like Loehle) so that the BCPs could still produce a HS shape.
At first the feeling around here was that perhaps there was a fertilization effect which created the growth spurt in the BCPs. Now that we’ve been privledged to see some actual strip-bark BCPs and what sort of results they give, I think most of us here are convinced that the strip-bark BCPs simply produce thick rings (or more properly, arcs), by their very nature. IOW, it has nothing whatsoever to do with temperature.
JEG said of Loehle:
I don;t have any difficulty with this question and agreed that Loehle should provide this information. But when I ask JEG whether such information should be provided by Mann, Jones, Juckes, Moberg – he gets all coy and says that was before his time. Not good enough.
JEG’s only consistent answer is to say that whatever standards he sets out for Loehle, he is honor bound to apply the same standards to Mann, Jones, Moberg, etc and let the chips fall where they may. It’s not good enough to be coy. He may argue that the standards did not exist at the time and that no blame could be attached to publishing these papers. That would be a plausible line of argument.
But he could not argue that, looking back, and knowing what we know now in the form of JEG’s objective standards for pseudo-science, that these earlier studies were not also pseudo-science. That’s where this is going.
This is what I wrote:
I never said I lacked the statistical or scientific background to understand much of what is posted here. Good Lord, if that was the case, I’d give up now. I said I was not up to offering a thorough critique at the level of a statistician or climate scientist of the statistical methods or climate science specifics. That is quite different from understanding the basics well enough to communicate them to others.
#118. ?You say:
That’s certainly not my position. I have no problem with JEG or anyone else critiquing Loehle. Ive endorsed some of the criticisms. However if JEG criticizes Loehle (for example) for not reporting a verification r2 statistic, then I am surely entitled to observe that it is hypocritical of him not to criticize Mann (or Juckes or others) on the same count. Or for any number of other misdeeds.
I used the term hypocritical advisedly. Being hypocritical doesnt mean that his criticisms of Loehle are wrong or that they should not be made. As to JEG’s failure to acknowledge the validity of similar problems in other studies, no one can force JEG to grasp the nettle of observing similar problems in papers by authors that he supports. Not doing so may be an entirely pragmatic course of action for someone in his position. But if he goes that route, we are surely entitled to say that the failure to make such acknowledgement is hypocritical and you should surely agree with this.
126 — thanks for the background. I’ve only recently read the Wegman and NAS material (as in a couple of weeks ago — I’m that green) I’d be very interested in reading some of this work. Can you point me to it? Is there any published literature? I am currently reading up on the different proxies, and so this would really help.
129 – my question would be is the criticism valid? If so, it is valid regardless of whether it also applies to Mann and whether the critiquer owns up to the fact. If the criticism is valid for Mann as well as other reconstructions, it’s context for a critique of the entire field. When asked if this applied to Mann and others, Dr. Emile-Gaey could have agreed if he felt able to do so — if he had read the past papers and recognized the lack in them. It might be entirely possible that he was unaware of the lack or had not even read that paper, which is now 10 years old. When I write papers, I try to keep as up to date with the current literature as possible. One of the very first multi-proxy reconstructions is like ancient history to those doing their PhDs now, I would suspect.
It seems rather like a litmus test to me. YMMV.
Susann: you need to appreciate the irony of an associate of Mann’s coming on this blog and flaming a paper for violating scientific standards that Mann and others have not met.
I want to add that I have not read every paper of my dissertation advisor, although I’ll never own up to that fact. 🙂
It is ironic. Sometimes, I’m too serious. 🙂
AGW and politics are inseparable. The reason is simple. If one takes the alarmist position, it leads to numerous restrictions of the individual, either from mandate or imposed economics (taxation). Many are on the table: changes in diet, changes in overseas travel, cars less safe or slower, foodstuffs from much closer at hand, perhaps restrictions on indoor climate. Some have been adopted. But, the freedom to undertake activities is under assault.
Craigs paper may have aspects that could bear improvement. He may have received more suggestions than he expected and in tones more painful than he imagined. But, at the end of the day, the implication of his paper is not to remove choices available to people today through future political action. Susann is correct: if we are looking at Craigs paper in isolation, then whether someone else has used the same approaches, tools, techniques really is not important. Passage of time often requires a more rigorous analysis from a later contributor, even if we are comparing to entries in the contest from 2007 Juckes, et al or 2006 Hegerl, et al.
That being said, I think the bar should be much higher for Mann and those whose research on paleoclimate proxies may serve as the fulcrum to lever through policy changes that will dramatically reduce current freedoms of diet, travel, interior climate, availability of medical care and a host of other aspects of modern life. Their research in substantial part is offered as the basis to change current freedoms, and if their research is put to that end (as it has in AR4 and elsewhere), then the most rigorous available analysis should be applied to it. Paleoclimate studies are critical to arguments concerning the present, as in at least (a) are current conditions other than within the historical range of normal, and (b) do the current GCM models function well enough to explain the drivers of historical events (assuming they exist) such as MWP, RWP, LIA and the rapid changes associated with such events. Frankly, without analysis that has passed the most rigorous acid test on these points, it will be difficult to ram through dramatic changes for everyman, as in Eat local, think Medieval.
I sense in the assembled crowd a respect for JEGs talents and intellect, as well as a hope that someone with that combination from the outside (or the inside, as the case may be) will hold to intellectually rigorous analysis theories leading to imposition of radical restrictions on personal liberty. Frankly Steve, its why I appreciate your work (particularly Waldos adventures as relates to historic temperature records). And Craig, keep ploughing forward and do the best that can be done with the data available. Use every constructive suggestion to improve the product, and let the effort take you wherever it leads. I am less impressed with the money mentioned repeatedly as riding on AGW generated policies. For me, the issue is freedom no one should restrict it without the most rigorous and airtight reasoning, and even then maybe not. Arguments to restrict freedoms should be revisited repeatedly, even daily.
Nor have I, but I certainly have read the most prominent papers from the field in which I study. Mann’s, while obviously not the best, are certainly among the most prominent.
Mark
Re #118: Susann,
I have been following CA daily for at least three years now, posted occasionally. It seems to me that for whatever reason, you are coming to an incorrect conclusion.
It seems to me that the issue is that the Mann et al corpus has been examined in detail here over the past few years, and found wanting. Mann et al have generally chosen to be obstructive, and not engage with serious and genuine questioners. That is how it is, and we can’t change that.
The commitment of most posters here is demonstrated to be the pursuit of sound science.
I am sure that Dr Loehle’s paper will be subjected to just as demanding scrutiny as Mann et als have been. If it survives the process, thats good, but it may founder on the same rocks that Mann et als work has foundered on.
It is clear, as you have pointed out, that the scientific process has been politicised. And under those circumstances, it is not surprising that we see “obscurantism” on the part of those who have been effective in influencing the electorate (70% of the population in Australia think that AGW is the most serious problem facing the planet) which in turn puts governments in a difficult position.
What is needed is a commitment to sound, fearless and robust science. Dr Loehle is clearly committed to participating in that process, regardless of the outcome, and, I suggest, so are most posters here. That commitment requires open-ness, disclosure of data and methods, and free discourse from informed people commenting on the issues as they arise. I think that you are seeing that here, although not always do the participants phrase their comments carefully enough – including this poster!!
The comment that Gore got it right in AIT simply illustrates a general lack of knowledge about weather and climate. A judge identified 9 errors, Lord Monckton found 35, and I have found several more. It is my experience that many people working in one area of climate or working on climate models generally have a poor knowledge of climate and climate mechanisms. The tunnel vision of specialization is evident in the IPCC reports.
You’ve been coming her daily for 3 years — what goes on here is normal for you. I have been here for a much shorter period of time and have visited far less frequently. It is all new to me. My perception is my own — right or wrong — who can say whose perception is right? Maybe neither, maybe both.
And that’s more than enough profundity from me for the time being. Good night!
If the data collection was paid for with public funds via a grant from an agency like the NSF or NAS, then the data belongs to the government agency and by proxy to the public at large. So yes, in some circumstances one can “rudely demand it” while being on the correct side of the issue.
Indeed, most of the publications these studies appear in require archiving. It’s not just a birthright.
Mark
Susann
JEG’s criticism of Loehle’s paper is in most parts valid and most of us in here also says so. What really hurts my eyes is that JEG immediately put the label of “pseudo-science” on Loehle’s paper even though (almost) all of the papers in this field has the same problems as Loehle has. So why is Loehle’s paper pseudo-science, but other author’ papers (Team members mostly) not?
It is very heavy criticism to say in scientific circles that someone’s paper is pseudo-science. And now JEG is making this accusation very lightly. Because of this behaviour of JEG’s many people in here tends to compare Loehle’s paper to Mann’s paper because that is the most famous in the field. Please, do not accuse (most) of us in here to look at this matter through coloured classes because that accusation just shows that you have not been here long enough to see that this blog is not as denialist-oriented as you often claim.
As a non-scientist reader of CA since early this year, I’m a little disappointed with recent the treatment of Dr Emile-Geay. I completely support Susann’s points, and believe we should welcome participation of this sort, not continually and repeatedly challenge his valid points because others do not meet the same standards.
Dr Emile-Geay made criticisms of Loehle 2007 that are completely in line with the thinking behind a lot of CA, and he agrees with the need to provide open access to data and code. Whether these criticisms are ‘fatal’ or can be fixed by editing is open to discussion. He has not directly criticized Mann’s papers or others, though in the context of his current role his refusal to defend these papers speaks volumes.
On the other hand, his defense of recent papers can and should be challenged. His use of the term “pseudo-science” was ill advised, not least because it can rebound, as this discussion has shown.
I view the involvement of Dr Emile-Geay, and the possibility of agreement on the necessary areas for improvement in the Loehle paper, as one of the first signs that the concerns of this blog are beginning to influence the climate science main-stream in a positive way. Once we have consensus on the strengths and weaknesses of the Loehle paper, we can move on to reviewing whether other papers meet the same standards.
PS: Apologies for using ‘we’ in this comment. I’m not including myself in the great work done here by others, it just made it easier to write and I hope read.
#143 I concur. This is my bottom line. If JEG is going to call Loehle (2007) “pseudo-science” – a very serious epithet – then let him brand all studies of that ilk with the same language. My review was as critical JEG’s, but I would never use that phrase to describe the work. Aside from being vaguely accusatory, it is ad hominem. And I’m still not exactly sure what he means by it.
JEG knows that the bar for publication is novelty/innovation, not correctness. Past publications such as MBH9x got a pass back then just because they were novel. They would not pass now. They were far from correct (and the researchers’ behavior was very far from accountable). Irony has nothing to do with the standard to which JEG should be held. If he is going to use fightin’ words to describe Loehle (2007) those same words must be used to describe the accuracy level of the work that is STILL the foundation for IPCC Ch 6: MBH9x, Moberg, Esper etc.
So, JEG, what is your judgement? Which studies are “pseudo-science” and which are not? There are many studies in the last 10 years which we thought were good science but which it turns out were absolute junk. Honest mistakes? Or pseudo-science designed to deceive?
IMO this matter requires resolution. Explain yourself, or retract the statement. I don’t see the banter ending without a clear resolution. (I think that, in part, is what Susann is fishing for.)
While we are at it I would also like clarity on the attack on the journal EE. As JEG & Dr Curry well know, a good portion of the papers in the other literature are invited submissions, and they do not receive nearly the same scrutiny as contributed pieces. How is EE‘s two-tier policy any different? Again, double-standards. Accusations unsupported by fact.
Contrasted with JEG’s accusation of “pseudo-science” is this other bit from JEG’s review from the Loehle thread
It seems slightly contradictory to suggest that “pseudo-science” should be published in scientific journals, but I think he is trying to say that the Loehle manuscript needs some work if it is to stand up to intense scrutiny and be used in a policy arena.
To my way of thinking “pseudo-science” does not become “science” overnight. The difference between them is more than just presence/absence of a map and confidence interval. For example, when Judith Curry failed to report confidence intervals in her BAMS article on hurricane frequency statistics (Mixing Politics and Science in Testing the Hypothesis That Greenhouse Warming Is Causing a Global Increase in Hurricane Intensity), I did not refer to her work as pseudo-science.
Dr Emile-Geay, I am confused about your position on the Team’s work. You insist that you do not endorse it and therefore should not be asked to critique it the same way you critique Dr. Loehle’s paper. Yet you are teaching a course on the centerpiece of the Team’s work. Do you start each class by telling them the Team’s work is sufficiently flawed to qualify it as “pseudo-science”? Is the purpose of your course to alert otherwise unsuspecting young minds to the flaws in the Team’s work? Or is your purpose to become one more voice preaching that the Team’s work is “established science” and that “the debate is over”?
JEG, Susann, bender et al,
As I sit back and review the conversation here, and the events of the last few days/weeks… I’m sensing that we’re experiencing a sea change.
The past: Mann et al, “The Team,” were more or less invincible. A closed system of data, publication and influence over the AGW issue, with surprisingly low standards. McIntyre et al were serfs at the door, trying to get their knocking heard. It really WAS all about Mann and the Hockey Stick, particularly in a political sense but also scientifically: they set and “controlled” the standard… to the point that even Wegman wasn’t really influential.
The future: solid science returns to the fore. Scientists (and policymakers of integrity) in many disciplines, plus a new generation of climate scientists, have recognized that good answers come from good science, and good answers are more important than politically correct answers.
Bottom line: we’re in a transition from Mann as reference point to good science as reference point.
JEG and even more Susann are evidence of this shift.
In the past, we had no choice but to compare new work with Mann.
In the future, Mann will be irrelevant except to the extent that he can produce good science.
Today, we’re a bit conflicted: we still compare to Mann, but we also compare to basic good science.
To use the old (true) saw about the US Secret Service and counterfeit money: to know good money you need to see both good AND counterfeit.
We’ve certainly seen plenty of garbage climate science. Let’s take this opportunity to rapidly help identify and/or set a high bar for good climate science.
As far as I’m concerned, anyone who cares about good science is welcome to help, even if they were or are identified as members of The Team. For this, I think the old cliques are irrelevant and best ignored.
Once we’re there, we’ll have a nice broom to take care of all kinds of stuff that needs to be swept away.
How about a deal: you tell us how exactly the MBH99 CIs were calculated, then I explain what I think of the matter. You know, before I know exactly how MBH99 CIs were calculated, I can not say for sure if there is any better way of handling the thing 😉
Susann in #108
You seem confused about why Mann is brought up a lot in this thread about Lohle, but then you write this:
Current political will is shaped by Mann and the “Team”, (AIT, IPCC).
Critiquing Lohle doesn’t change political will, (currently), critiquing Mann might. I don’t believe that McIntyre is “against” AGW, (I actually am not sure of his position – which is good), but I do think he’s against basing political will on the “Team’s” flawed work.
Patrick M.
In an attempt to get this discussion back on track, so researchers like CraigL and JEG don’t get frustrated/bored and disappear:
It is a really good thing for CA (and arguably for the science) to have CraigL and JEG here. It seems to me that both are interested in moving forward from (and past) the MBH papers and the critiques/defenses thereof. CraigL has put a new paper out there, and JEG says he would be further interested in in a discussion on Mann et al. 2007 (new paper). It seems counterproductive, if the objective is to move the science forward, to continue to rehash the MM/MBH hockey wars. I know that the hockey wars is the key heritage for this site. But if you would like to attract (and retain) researchers like CragL and JEG on this site, I wouldn’t focus on rehashing MBH, and make bashing MBH a litmus test for engaging with a researcher who decides to drop in. The main point is that science moves on. And researchers are interested in moving forward.
And it is really interesting to hear the perspective from people like Susann, who are newcomers that are trying to follow this. Logic and interest from an attentive newcomer helps freshen our perspectives.
If someone wants to hash out issues related to peer review, impact factors, bias for journals, etc., I would be interested in engaging in such a discussion when i have some spare time. I think this thread is not the place for a more extensive discussion on this topic.
Steve: Judith, your suggestion that we discuss Mann et al 2007 is hardly an invitation to step outside of the Hockey Stick wars, as it is a Hockey Stick paper. I will turn some attention to Mann et al 2007 (which still relies on Graybill bristlecone chronologies as its active ingredient) on another occasion. For now, I note that in this new paper Mann argues one more time that he get a HS without his erroneous PC method. However his network notably has kept all the Graybill bristlecone chronologies. We did not say that Mann’s PC error was the only way of accomplishing the parlor trick of sawing the woman in half; it was however used in MBH98. We observed that the reason why the trick worked was because the bristlecones. In MM2005 (EE), we noted that the bristlecones naturally dominated the AD1000 network through longevity; that the issue was not merely mathematical artifice. I don’t know why this is so hard for climate scientists to understand. In Mann et al 2007, we have another complicated multivariate method using a network dominated by bristlecones in its early stages. The word “bristlecone” does not occur once in Mann et al 2007 BTW. So your invitation to Mann et al 2007 is not one that is conducive to moving forward. Mann also argues that verification r2 scores don’t matter. Personally I’d be more interested in hearing a statement like this from a statistician who’s examined pseudoproxy networks, rather than Mann’s self-serving assertions. It’s always a lot of work to figure out what Mann is really doing and, if the issue matters to climate science, you really need to hear this statement made by Wegman or someone like that, not Mann. Given that CE and r2 statistics are linked, it’s amazing to see the difference in discussion of these statistics not just by Mann but also by the NAS panel.
HAving said that, I have been thinking about the Loehle network some more and will post some new thoughts on it.
Thanks, everyone! I have enjoyed observing the debate. JEG stimulating; SteveM characteristically rational and measured; bender and others challenging (apologies to those I have not listed).
FWIW: I share JEG’s view that Loehle’s paper is substantially less than complete. At first, I thought Loehle might be engaged in some sort of Sokal-esque hoax. (To be fair, the same thought (e.g. here) crosses my mind frequently with respect to Mann, who is obviously far too smart not to realize the absurdity of some of the positions he takes).
If the stakes associated with climate change were lower, it would be tempting to develop the Monty-Python possibilities this debate offers — to transgress some boundaries — just for the chuckles.
😉
#150
Error elimination phase is needed before we can move to more interesting problems.
Re Bender’s #74,
Or of the European Science Foundation workshop held in September in Frascati, “Econometric Time-Series Analysis Applied To Climate Research”? The ESF site is playing up at the moment but the agenda can be found
here.
It looks like someone is trying to do exactly what many posts in this thread say should be done.
Apologies if this has already been mentioned here. I couldn’t find any references to it (and was surprised not to find any).
@@Susann–131 & 133
One of the very first multi-proxy reconstructions is like ancient history to those doing their PhDs now, I would suspect.
Based on my experience, I would suspect it’s not ancient history. I have a Ph.D. in mechanical engineering. In physical sciences where experimental data can be slow and difficult to collect, and analyses take time, I doubt 10 year old literature relating to experimental data specific field would be considered ancient history.
Certainly, if the results of a paper are sufficiently widely disseminated, 10 years is not a long time. This is particularly so if the results of the investigation are being discussed in the US Congress.
I’d be surprised to learn even one Ph.D candidate studying dendro-chronlolgy has not at least glanced at Mann. That would truly exhibit the sort of lack of curiosity not typical of Ph.D. candidates.
Granted, I don’t work in dendrochronology or climate change. Maybe they are a particularly incurious bunch, or maybe paper become old and dated more quickly in dendrochronology than in multiphase flow. If so, that would be an interesting observation for your own study.
But, on more specific issue: Is it likely a Ph.D. working with Mann is unfamiliar with this paper, that seems to me extremely unlikely.
You suggest that people don’t read every paper by their advisors or collaborators. This is true. I also had not read every paper by my graduate advisor.
But I think that’s not the correct question to ask oneself. Were any of your advisors papers widely criticized? I, and every one of my fellow grad students had read the S.L. Soo papers that had been criticized as fundamentally flawed. And we all had opinions which we shared over beer. Also, when we met others in our field (particularly those who wrote the letters or their graduate students), they always asked us what we thought of two particular papers both of which were old before we started grad school. People our age asked us. People our advisors age asked us. There was no escaping having an opinion on these two papers.
Mind you, depending on the personalities of the graduate students, we all answered differently.
I suspect the fact that Dr. Emile-Geay will not defend that particular Mann paper speaks volumes. Since he made an appearance on a blog to discuss similar papers, he has made what he says, and what he won’t discuss, part of the conversation.
Susuann,
You might have a point that Mann is brought up too much around here. I’d say it’s generally for legit reasons.
But in specific reference to JEG’s critique of Loehle, it was JEG himself who mentioned Mann’s work in doing so http://www.climateaudit.org/?p=2380#comments, post #143:
So JEG’s original critique/review itself mentioned Mann at least 3 times. It’s perfectly relevant by itself to discuss JEG’s critique mentioning Mann’s work when JEG’s critique does the same.
Steve M wrote in the lead post,
I disagree, but rather still concur with Craig that R^2 makes no sense for his reconstruction, since it is a simple average of pre-calibrated series, and contains no regression-based calibration of its own. As I argued at #385 of the 11/15 thread (P-2380, “Craig Loehle Reconstruction”), he should instead have just computed standard errors using the conventional formula and a few words of caution about their interpretation.
Steve is evidently comparing Craig’s series to a global temperature index which he does not identify, perhaps HADCRU. This is fine and interesting, but goes beyond what Craig attempted in his brief paper. However, since Craig’s numbers are already in degrees C, rather than mere proxy Z-scores, the issue is not whether they positively correlate with direct instrumental readings, but rather whether they are equal to them, after removal of a sample-specific constant and taking their respective measurement errors into account.
This could be done by taking the difference between the two de-meaned series, and then computing the standard error of the difference as the root-sum-of-squares of their individual standard errors. Even HADCRU admits it has a standard error, which appears in its graphs and hopefully in a table. Craig’s standard errors (which he should compute and post on a permanent webpage of supplementary material for his article) will presumably be larger, and will increase toward the end of his series, where the number of contributing series declines.
Of course, since the means have been removed, the differences must sum to 0, so these are not independent (or unbiased) tests. Furthermore, since he has taken 30-year averages, the differences are serially correlated. One may as well take a 30-year average of the comparison series, in fact, as JeanS did at #5 above, to avoid the appearance of meaningful year-by-year fluctuations in the difference.
But the comparison is still worth making by those who want to extend Craig’s paper in useful ways. I would hope that Craig would post links to such 2nd party extensions on his Supplementary Materials webpage.
Jean’s calculation at #5 shows that Craig’s series does match HADCRU quite well during their overlap period, but she does not compare the differences to their purported measurements errors as I propose.
A further word of caution about conventional standard errors is that they assume that the errors are uncorrelated. In Craig’s context, this means that each series reflects global average temperature, plus a local idiosyncratic error that is uncorrelated with that of every other of his series. If all 18 of his series were located in Rhode Island, this would not be true. Hans’ map shows that they in fact have a remarkably good geographic distribution, but any future study should keep an eye on this problem. Climatologists try to get around it by “gridding”, I gather, but that is rather primitive in comparison to modern spatial autocorrelation methods (which I don’t pretend to understand).
Of course tests such as those based on R^2 are pertinent to the calibrations in the papers Craig uses, and someone should review the validity of these studies someday. R^2 can be used to compute the “Regression F Statistic” which tests the hypothesis that all the coefficients other than the constant are 0, but only provided there is no serial correlation in the errors. In order to be valid, any serial correlation must first have been removed (eg by fractional differencing for AR(1)), or at least measured and incorporated into the coefficient covariance matrix. In the latter case, the Regression F Test can still be performed by standard methods, but R^2 will no longer give it.
For those people who posit that JEG may be unfamiliar with MBH98 as merely one old paper, please note that he is presently teaching a course on the Hockey Stick – the syllabus to which is online here . The failure of MBH to report verification r2 and CE statistics – which he demands here – can hardly be unfamiliar to him. Further the course covers the Wegman Report and the circumstances leading up to it.
No one here is holding JEG responsible for the past, but if he’s teaching a course on the Hockey Stick, he shouldn’t profess complete ignorance of the verification r2 issue.
When Result(Y), different from Result(X), is interpreted as rebuttal to Result(X) or at least calls into question the exclusive validity of Result(X) then the data, methodology, and acceptance of Result(Y) is relevant relative to the methodology, and acceptance of Result(X).
To apply more restrictive standards to Result(Y) would be unfair in the context of implicit comparison to Result(X) and hints strongly of agenda. The simple fact that comparable datasets and equivalent methods produce Result(Y) instead of Result(X) calls that result into question regardless of whether any problems are in the data or methods. Just as producing the hockeystick signal using random noise as data and the same method raises legitimate questions about that result regardless of the details.
JEG said here:
Boo hoo.
It’s upsetting to me as well – that’s one of the primary purposes of this blog.
JEG, if you’re going to raise this issue, then let’s look at low-grade statistics being sold as climate science. Mann et al 2007 is not even mentioned in IPCC AR4. Sure we can look at that, but you can’t just exclude from consideration every study mentioned in IPCC AR4 merely because Mann has “moved on”.
Off topic and I apologise in advance, but
As an interested (very)observer of this site, I would like to make a small correction. The quote is “Lay on, McDuff… meaning, start hitting as with swords, so perhaps this is not the right quote. (I hope)
Susuann,
Im happy to see Manns work referenced when examining Dr. Loehles work so lead on, McDuff!
Sorry, this was supposed to have been quoted
If I put on a policy hat, and started thinking like a policy wonk, I’d be asking the exact same questions Susann is asking, which, if you take a close look, are the same questions a lawyer would be asking. In that mindset, the Loehle and Mann papers are two separate issues, and any hypocrisy is yet another issue.
What I think is going to be a problem with that approach is that it will run into the same problem that courts run into: they can’t analyze the science itself, so they try to infer the state of the science by analyzing the credibility of the expert witnesses. But a good court will also try to get a sense of the sincerity of the expert witnesses, and if one uses diffrent standards for different situations, that does detract from that expert’s credibility.
I think Susann is looking for certainty, which is what any court would look for. The problem in this case is that political proclamations notwithstanding, there is no certainty, only degrees of uncertainty. The forefront of the science is about getting the best estimate of the uncertainty. The best, with the current state of science, that we can do is to get a reasonable estimate of the uncertainty. That’s what the controversy boils down to. Policy is going to have to be decided based on the best estimates of the uncertainty. That’s all we’re going to have to work with for the forseeable future.
Re confidence intervals: I know how to compute ci when all the data are annual resolution, but what about when the data are sparse in time and have dating error? No one has studied the dating error issue except in my 2005 pub (cited in my paper), nor the problem of sparse data. The ref JEG supplied (li et al) does not deal with either issue. This is why I did fig 2 & 3. I throw this down as a challenge to those who think I am not so smart–give me a reference that shows how to compute ci under these conditions.
Dr. JEG.
Dec 3 is the last class? How about a guest speaker. I have one in mind.
Steve:
To be fair, I’ll point out that Kim Cobb gave the lecture on that part of the course.
[Steve: I agree with your point and have edited the previous remark accordingly. I’ve snipped this comment only to avoid repeating the inflammatory and unecessary remark.]
Yes, reading Susan’s #117, it seemed to me at the time vaguely out of character for MacBeth to say such a thing. The actual quote is: “Lay on, McDuff, and damned be him who first cries ‘Hold, Enough!'” (Macbeth V;7)
In Susann’s defense, the line is widely misquoted.
Lay on, Steve!
To be clear, I advocate the discussion and dissection of MBH 98/99 and think it is necessary, given the reliance of the world on it, I just suggest that you can’t forcibly drag someone into defending it. Continue discussing it, without JEG, and allow JEG to comment on the newer papers, if that is more comfortable.
Your musical interlude
Allusions to present company are not intended
re166: Steve, thank you, this shows very admirable character. I’d suggest removing the first two words after your quote in 159 would similarly change that comment to be a very different tone that is constructive rather than destructive.
It does not seem to me that JEG is called to defend MBH 98/99, only that in the context where Loehle’s results, as interpreted by third parties, call into question the exclusive validity of MBH the same standards apply to both.
When the issue is comparison of results, using the same eval standard for method is only fair.
#153
Vinny, the post is here:
http://www.climateaudit.org/?p=2091#comments
(managerial coverup imo)
# 153
Sorry Vinnie,different workshop,my bad.
JEG said:
My questions are:
If you do co-author a paper with Mann, will you insist that this new paper meet all the requirements that you’re holding Loehle to (i.e., will you insist on the verification r2 test)?
Will you insist that the data and codes be publically archived?
Will you question the proxies used?
You’ve said that you shouldn’t be questioned on Mann’s old work because you weren’t around then.
Now that you are aware of Mann’s problems with his earlier papers, what will YOU do to insure that this new paper can stand your rigorous review? Remember, it will be listed as Mann et al. Your name would be forever linked to this paper, good or bad.
When scientists (including the “et al”) see that their papers are being read and reviewed by others outside their “clique”, they might try harder to ensure they can stand critical review.
This might be a way to ensure that the science is correct, no matter what the outcome.
A lot of people are asking JEG whether he will ask for r^2 stats in Mann’s next paper. I think they are misinterpreting JEGs point of view (or I am; not sure which!)
JEG’s comment regarding r^2 in Loehle was (IMHO) more of a cheeky snark at ClimateAuditors than a serious request for r^2. If you look at post #63, particularly paras 5,6, JEG is clearly of the view that r^2 is not important to a reconstruction of decadal based data. He asked why r^2 is important and I’m not sure anyone has answered his point here yet. I think his position is more like “if climateauditors think r^2 is important, why aren’t they demanding it” rather than “I think Loehle 2007 should have calculated r^2”. The nuance is subtle but important (and of course fails as CA has requested statistics for Loehle 2007, and indeed calculated some already). This means the claim that JEG is being hypocritical may not be on sound ground IMHO. I would argue, instead, that JEG had an incorrect perception of climateauditors.
I’ll have a go at answering para 5/6 of #63, although I’m not a statistician, and others would do better.
JEG cites the Wahl and Ammann paper (figure S1 and frequency component argument) against r^2. Firstly, as a matter of general statistics, you should never assess statistical tests with anecdotal examples; particularly not extreme anecdotal examples with extreme inhomogeneities in the data or residuals such as illustrated in figure S1. They should be tested through a model and distribution (or Monte Carlo methods).
More pertinently, r^2 is not just a test for how well the reconstruction looks. The frequency set that r^2 works with is a similar set of frequencies to which the regression step operates. If r^2 is finding garbage, the regression step is probably operating on garbage. If the regression step has worked well, and the proxies are well behaved, then the calibration r^2 will be good, and similar to the verification r^2. A high cal r^2 and poor ver r^2 tends to indicate that the regression step has failed, and the regressed result is merely a meaningless overfit of the data to the model.
So the r^2 test is not really saying “does the construction look nice”, more asking “have the statistics done what they were expected to do, or have they gone off the rails somewhere?”.
As for RE, since the test is heavily dominated by just one single degree of freedom, it is a weak test, and can be easily “tuned” for (through data selection, or by, for instance, botching an offset into the methodology), which is why it is insufficient evidence that the reconstruction is good. I need to remind myself how CE works again to comment on it, so I’ll leave that to someone else 🙂
JEG also demanded Loehle 2007 should calculate RE/CE. I cannot see how RE or CE could be produced as there is no explicit calibration / verification period, so the request for these parameters are inappropriate for this type of reconstruction.
Susann said in post #117 in response to my post:
No, I am only reacting to your apparent litmus test that Loehle shall be discussed and analyzed without mention of Mann and the hockey team and their sometimes misapplication of statistics. I would say let us drop this insistence, Lady Macbeth, debate it no more and let the games began — but on further notice the games have began and appear to be progressing at a rapid pace. Shall we join the crowd, my Lady?
Re: #175
Spence_UK, thanks for your comments on JEG’s ambivalent references to the use of an R^2 statistic in temperature reconstructions. I was of the same mind as your comments indicate and wanted to see a direct answer on this issue but felt relatively less qualified to reply.
The Duke and Jeremy:
“Lead on, McDuff” — “Lay on, McDuff” — doh.
Obviously not a Shakespeare expert!
Either way, I believe I wrote that in response to the notion of reviewers addressing shortcomings of Mann when discussing Loehle, so it still would apply, no? Lead on suggests that the other person begin the journey while lay on suggests the other person being the attack.
I will repeat that I think it is fine to discuss/critique Mann in connection to Loehle, but I disagree with this notion of forcing Dr. Emile-Geay to do so.
As to my defense of Dr. Emile-Geay and suggestions that perhaps he had not read the papers or recognized the flaws if he had read them — I see he should be quite familiar with the papers, given he is teaching a class in the hockey stick. 🙂 I do believe, upon re-reading his review of Loehle that he was throwing that demand back into the face of CA members, rather than really meaning it.
I would love to be taking that class. I have downloaded the syllabus and will use it and the referenced papers as a guide to understand the debate from his perspective. I think I will develop a directed reading course of a similar bent that will cover this ground. I will use this blog as the source material for the “other” take on the topic. 🙂
Regardless of whether Dr. Emile-Geay has read the papers or not, I was giving the man the benefit of the doubt to be polite. My father taught me to assume that other people were honorable until proven otherwise.
I was not insisting that Loehle be discussed without mention of Mann, only that a critique of Loehle can be valid without reference to Mann, and that insisting that Emile-Geay indict Mann like imposing a litmus test on reviewers.
changing centuries
The game’s afoot. 🙂
Not so — I think I have a good sense of the uncertainty of the science, but even given that uncertainty, one has to plan for as the old saying goes, those who fail to plan, plan to fail. What’s important to me is clarifying the issues, points of contention, main limitations in the science, the parties involved (“stakeholders” as we politely call them), the options, potential risks, benefits and outcomes of taking this or that action. Policy analysts always have to work with the best that is available, even if that isn’t perfect or completely certain.
Agreed about the effect of labelling a paper “pseudo-scientific”. I don’t believe I have often claimed that this blog is denialist-oriented. I don’t believe I have claimed that even once.
Re #183
Thank you for admitting that labelling a paper “pseudo-scientific” requires some serious evidence. I think that in the absence of that evidence those kinds of statements should be retracted with a formal apology. Sure, it’s a rookie error to leap to that judgement. But in this case it’s a serious error.
@Susan–
Would believing Dr. JEG had not read the papers be giving the benefit of the doubt? On the one hand, believing him might clear him of the accusation of telling falsehoods. On the other hand, it would be indicting him of unfamiliarity with important literature in his own specialty, which, I think you would agree, is rather a negative thing for Ph.D. doing research. Non?
I agree one cannot force Dr. JEG to give an opinion on Mann’s paper if he does not wish to do so. But his failure to comment must give us pause…. So, his answer is devoutly to be wished.
Lucia, I have no allegience here, although I appreciate the tolerance people have shown me. I feel no burdern (or benefit) of membership in either community, whether CA or RC, nor do I feel the need to defend one side or the other (if I do so, it is because I think it is deserved), nor do I bear the weight of past slights and injuries and resentments towards one side or the other as so many here and there appear to. So, I looked at what was taking place in the response to Emile-Geay’s review and called it as I saw it – a partisan attempt to get Emile-Geay — a known member of “the Team” — to do something that was unnecessary to the issue of giving a competent review of Craig Loehle’s paper. What may make complete sense to regular members of CA does not necesarily make immediate sense to outsiders who do not share the same views and experiences. The point of validation and verification is not to extract confessions from heretics or exact retribution from the “opposition” but to further the science, or at least, I would think. If the point of verification and validation is to score points on a battlefield, the matter is completely different.
As to whether it is an insult to assume a recent PhD has not read all the “relevant” literature — I suppose it depends on the discipline and what is considered relevant. Some fields move very fast and so a 10-year old journal article in a related but not identical field might be considered too old to be relevant to current research. Or not. I’ve taken classes where people were still reading Herr Freud and in one genetics course, I read McClintock who was by that point, ages old. In the field where technology is a large driver of science, where new technology renders old research archaic, I would not expect that students would read the old stuff. It really depends on the discipline.
Also, I tend to have a very strong devil’s advocate streak in me. 🙂
186 susann
Students writing PhD theses had better read the pivotal literature in their field, even if it is 30 years old, and the Mann hockey stick paper is pivotal, whether you agree with its position, or not.
@Susann– My impression is you are here to study, right? I’m new myself. I’m fascinated.
I agree that people are probably trying to get Emile-Geay to discuss things he doesn’t wish to discuss, and no, he’s not required to discuss that paper, nor is he required to defend it. He didn’t write it..
My only point is, it’s natural for people to ask themselves, if he doesn’t want to engage in the ongoing conversation, why did he pop in? Pro-actively dropping by a blog isn’t quite like being sent a manuscript to review for a journal, is it?
My Ph.D. is multiphase flow– and as you saw I commented earlier. To me, 10 ten years doesn’t seems all that old. Not old at all in fact– especially is a paper is good, introduces a new technique which proves useful and survives the test of time. Of course, 10 years is old if the paper turns out to be seriously flawed; then one ceases to speak of it. Right? 🙂
Still, I think out of curiosity, you might need to check the average age of papers cited in a premier journal to decide if 10 years old is archaic. It’s probably useful to also compare those discussing experimental data to theories. My husband subscribes to BAMS, and I just noticed one article in the August 2006 issue cites a paper from 1620!!! Imagine that?
Judith Curry writes:
Mann et al 2007 is a discussion of pseudoproxy performance. In May 2006, I did a similar sort of analysis on Echo-G pseudoproxy networks from my own viewpoint here.
Judith should read that post referred to in #187 because your wavelet decomposition scheme would be useful in her attempts to isolate the various high/mid/low frequency components of hurricane occurrence vs. climate data. The 5y and 10y periodicity in hurricane occurrence would go some way to “explaining” (i.e. accounting for) the low count for 2007.
Susann
I do apologize I made a hasty conclusion of you to point in your posts between my posts of 104 and 142 that most of the people in here only focus their criticism on Team papers. Thus, I made a false conclusion that your “tone in your posts” also made a claim that CL belong to the denialist camp.
I meant no harm and your criticism is valid.
My do apology my harsh words!
Woops…CL should of course be CA.
Lucia, given what I now know, Emile-Geay’s reluctance to discuss MBH is completely understandable. 🙂
One might be familiar with a paper from “the old days”, but might not have subjected it to the same scrutiny done to a new paper, with fresh PhD eyes (I don’t have them yet — I’m still at the embryo stage). Also, really, what is being done at CA and in the blogosphere is very new, don’t you think? Climate Science for the rest of us unwashed? I think it’s high time. And, knowing human nature and scientists, how else can we expect those in mainstream climate science to respond?
Emile-Geay’s tone and reticence to discuss Mann is completely undertandable, if not also inadvisable for someone hoping to be taken seriously here. But as another outsider, I did feel people were trying to force him to do something entirely unnecessary. Steve M posted the work here and opened it up for serious critique. Emile-Geay stopped by and let loose, admittedly in a dismissive tone, but did deliver a substantive review which Loehle found useful. I think it was a worthy venture.
Now, I want to see Mann’s new work subject to such a review.
One might avoid it like the plague. 🙂
Give me a break. His tone was arrogant and his review was redundant. And now his is trying to quietly back-pedal away from his ridiculous assertion that Loehle (2007) is “pseudo-science”. This indicates unbalanced, rookie judgement, and he admits as much on his blog. So why not apologize for the remark and submit a formal retraction?
191, 192: It looks like he discusses the papers with his students. Wonder what he says to them?
re: #194 jae,
I wonder if he’s trying something like Dr. Curry did last year. In her case she was “exposing” (“inoculating”) her students to CA, possibly to see if their vaccination had taken. Don’t know if JEG has something similar in mind or not.
Looks to me like JEG’s a real scientist, actually cares about the truth, doesn’t mind mucking it up with either side as long as it leads to insight.
re 196
What is going on here is quite significant. It is an example of open source science. One thing that is critical to the success of open source projects is a sense of respect among the developers both between themselves and, importantly, to those coming in from the outside. It is too easy for a group to become insular and self-satisfied. People involved in this issue need look to far to find an example of that promoted by the UN. If someone comes in with views that challenge the prevailing set of comfortable ideas in a group then it could be very profitable to listen to them.
This is a very long video from Google about the subject. However it does bear watching
@Susann–
Yes. Science discussed in such great detail in the blogosphere is new. Scientists have jumped in with both feet. I’d have to check which of Mann’s RC or McIntyre’s CA are older (or which is the oldest climate blog), but I think it’s fair to say that if a scientists do decide to jump in at blogs, or create blogs, it’s fair for a wide range people to have opinions and express them either at blogs or in comments.
It’s also fair that many points of view about questioning blog visitors be presented. On this particular thread, which is about what’s fair when criticizing papers, I think it’s also useful to describe a variety of points of view vis-a-vis critiquing scientific papers. In that regard, I’m commenting on something you said about the critique of Lohle’s paper
In some sense, I agree with you: Yes. JEG isn’t required to comment on Mann when criticizing Loehle. No one can dictate what JEG choses to consider or not, and it’s fair of you to point that out.
Still, in the scientific tradition, it’s only partly true that a valid critique of Lohle’s paper stands independently of any problems in Mann. A shortcoming is a shortcoming. Lohle accepted the critique in this light, and Steve M. concurred with these shortcomings. So, clealry Lohle, Steve M, you and JEG all agree on this issue: flaws are flaws.
Nevertheless, during the normal course of scientific peer review, the full critique of Loehle does not stand entirely independent of shortcoming of other papers in the literature.
When acting as a reviewer, one tries to identify shortcomings, but then considers whether or not a shortcoming in a current paper is shared by previous papers that, nevertheless, were thought to make a contribution despite that specific shortcoming.
Everyone accepts that papers are never perfect. Some issues are intractable with present abilities and finite resources. So, presented with a current paper, the ultimate question a reviewer might ask is this: Despite the recognized shortcomings, can people in the field learn something important from these results?
If the answer is “yes, people can learn something important”, the reviewer may recommend a paper for publication only insisting the author state the short-comings, mention them as assumptions, and possibly explain why correcting the issue is an intractable problem in the field.
For example: A computational model– say for example, one used to predict what might happen during a LOCA in a nuclear reactor– might use approximations that are still debated by various people in the literature. If every approximation needed to be fully tested and resolved before one could publish or even use a computational model one could never do anything. So, in this light, reviewers in multiphase flow, did assess shortcoming in the light of previously published literature. (You will note that if you switch the “nuclear reactor” code to a GCM, very similar arguments are advanced for use of GCM’s despite the fact that approximations that may be in doubt are used.)
So: when assessing shortcomings, many reviewers do compare and contrast them to shortcomings shared by other papers published in a particular field. They do make judgments as to whether that deficiency is such that it blocks publication even if the results would be of interest to the readers in the research community. Sometimes, they recommend publication despite that shortcoming.
In this regards, people in comments here are asking the same sorts of questions various some reviewers might ask (though, possibly not JEG.) Different reviewers may make different final judgments, but I believe these sorts of questions are asked by at least some reviewers.
(In my experience, these sorts of questions are also knocked around at professional society meetings, with various points of view presented. )
So, in that light, maybe, a reviewer (other than JEG) might respond to questions about how Mann’s paper relates to Loehle’s with something like this: ‘Mann used the standard technique available in its day. But, its deficiencies were identified by “Dr. X” and “Mr. Y”; since that time “Z” has developed a better method used today. So, even though Mann’s paper was acceptable 10 years ago (when, evidently dinosaurs roamed the earth), climatologists now believe one should use “Z’s” method or improve on it when doing similar sort of work. For this reason, Loehle’s paper should not be published today.’
(I would also point out that this sort of expansive answer is the rule after presentations at professional society meetings. “I am unfamiliar with my collaborator’s paper which is 10 years old” might provoke titters.)
Absent the traditional expansive answer to a direct question, those who read Mann’s 10 year old paper, notice it shares the very trait JEG criticized in Loehle and are left with no information about progress in the field, JEG’s understanding of that progress or JEG’s opinion of why the criticism is valid for Loehle despite having been acceptable 10 years ago.
And my observing this forces me to point out a corollary to your rule that JEG is not required to discuss Mann. It’s this: people are not forbidden from pressing JEG on these points after JEG takes it on himself to appear in comments at a blog and describes himself as an authority on these points. After he fails to answer, they are not forbidden from, well, speculating about his reasons for refusing to answer.
While I agree that it may seem like petty tribalism to focus so heavily on the flaws in Mann (or Hansen, or GCM’s or any of the other several topics often discussed on this blog), it might also seem petty tribalism to suggest that one may not discuss the flaws of Mann & etc. in order to get on with things or somehow get back on track (whatever that track might be).
The cultural issue may be which tribe one is in. But it’s not clear that prohibiting or editing discussions of Mann, or refusing to discuss them, is any less tribal or any more scientific than discussing these things often.
Worth saying twice.
===========
198, Lucia, again, I think it’s worth repeating that part of the problem that policy types have with this is that the scientific publishing system was never designed to drive policy. It was designed to get new and novel ideas and information out into the intellectual commons. Trying to use this for policy is perilous. If the intended outcome is policy, a different process is needed (thus my allusions to the court paradigm).
Funny the attitude of the academics who drop by on CA. The way they try to be nice guys (and girls), and admit that what is being done here is kind of cute, but of course it’s never “real” science. Now, we’re even told that it’s “pseudo” science. Of course, it cannot be “real” science because it’s done by people without the proper degree, and with no position in academia, the only valid passport to enter the land of scientific truth. How hard it is for an academic to admit that there is a world outside their little pathetic ivory tower, and that there is intelligent life out there! And when the going gets tough and they’re asked to take position on substantive issues, suddenly they always have that very, very busy schedule and it’s “well it was nice talking to you but I’ve got other so much more important things to do!”. Oh, and always complain about the unfriendly comments, like they have to read everything that’s posted here. Yes it’s a tough world out there, outisde their inner circle of certitude and self-congratulation. Especially in climate science, where they’re working so hard to save the planet! What ingratitude!
Allez, Julien, un peu de courage!
I prefer to separate those outsiders (we non-aboriginals who came after the original postings were once outsiders) who come here to join the debate (rough and tumble as it may be at times) and add to the knowledge bases of others and their own and those that come here to primarily comment and lend their opinion of what they view as a generalized POV and methods of rendering that POV of the blog as a whole. They are the rare posters who limit themselves to pursuing one or another of these activities, but given the nature of blogging it takes but a glance around the internet to see that open access blogs all have their own personalities if one wants to generalize, and given this character of blogs I find it an inefficient use of band width to spend so much time analyzing the posters and not the issues at hand.
I have experienced outsiders with an outsiders POV come to blogs to make their point in a polite and objective manner and gain the respect of the insiders and in a couple of cases become folk heroes to the blogs. They invariably avoided personality issues and stuck to objectively making their point.
Let the games proceed with full attention to players progress on the field of debate and while limiting our attention to the cheerleaders and other activities peripheral to the game to those infrequent time outs and lulls in the action on the field.
Lucia, I appreciate your insights. I have only attended one academic conference and presented one paper and even though it was very focused, the literature referred to in the papers on our one panel topic was very wide – not all of us had read the same papers or shared the same perspective. I accept that if you have years of these panels and conferences under your belt, the view of them may be different from my one experience. I am a neophyte in this area and in the whole graduate school world and so am learning a great deal. Thanks for your patience.
Still, I was taken aback at first by the tribal feel to this place (and the other).
Science is a bloodsport cloaked in collegiality.
@Larry– 200
You are correct. The peer review process publishing system is not designed to drive policy. Generally speaking, the peer review process is designed to permit ideas to bubble up, be disseminated and discussed and tested over time. When an area of investigation touches on important political matters, thing can get very dicey both for peer review, scientific process and policy evaluations.
Had politics not intervened, the issue of R2 would likely have come up within a few years of Mann’s publication. People would have recognized the problem (or, I assume this because JEG doesn’t seem willing to defend the thing and criticizes it’s flaws by proxy.) The paper and it’s graph would have fallen to the wayside as better papers without the shortcoming were published.
Peer review would have done it’s normal job out of the glare of the public eye. If AGW is true, and sufficient data exist, someone would have probably have done a rigorous analysis and still come up with a hockey stick with good R2 statistics.
But that’s not what happened. Is it?
(FWIW: I suspect the problem is lack of data. A great deal of empirical testing simply falls in the realm of high probability of beta error. I think if those– like me– who believe in AGW would just admit there are limited amounts of data more convincing cases could be made in favor of the theory. But hey, that’s just me. )
Susann,
I think the “tribal” vibe you are getting from this series of threads comes from the fact that the two sides of the discussion are coming at the issue from very different positions and with very different goals.
The CA side, represented by postings from Steve M, Bender, and a few others has been reviewing climate reconstructions using proxy networks for many years now, and they have discussed the pros and cons of nearly every reconstruction since MBH98. Within that context, Loehle’s paper presents something new both in content and methodology that has been missing from almost every recent reconstruction. If you look at the summaries posted by Steve M, there is very little new either in the content of the various proxy networks or the methods used to combine them and tease out a signal from any of the new reconstructions published by the hockey team in the last 5 years. They all used roughly the same methods and proxy networks and used statistical manipulations not found in any statistics reference to tease out a signal. So in that context, Steve M and others here are evaluating Loehle’s paper against the entire body of reconstructions and reviewing it to see where it fits in the landscape, as well as to determine if his methods are better or worse than those used previously.
On the other side, folks like JEG have come here to specifically critique Loehle’s paper, and are quite uninterested in how that paper fits within the landscape of other reconstructions. They are more interested in pointing out the flaws in Loehle’s work than taking on the philosophical question of whether the method used by Loehle is a better way to do things than the previously published studies. Thus you see him continually trying to redirect his focus back to the Loehle paper and ignore the implications of the other papers and authors.
With that framework in mind, I think the “tribalism” you see is simply a difference in focus. The focus of this blog is auditing climate science as a whole, not just one isolated paper at a time. If you look at the critiques of other papers that Steve M has posted, I think you will find that while he may delve deeply into content and methodollogy, he also ultimately looks at how each reconstruction compares to those that came before it and how the methods and data used were handled relative to other similar papers. In other words, it is a difference between JEG’s “small picture” focus on the Loehle paper, and CA’s “big picture” focus on the methods and methodology of climate reconstruction using proxy networks. CA often is accused of diving into the details, but if you read around here enough, you come to recognize that Steve does a better job than anybody I have seen on the net of properly placing his comments and reviews into a larger context of what they mean to climate science as a whole.
An unfortunate side effect of all of this is that Steve’s scrutiny of Hockey Team work is rarely received warmly by those authors, and there have been some truly epic battles simply to obtain the data and methods that should be published according to the policies of the journals in which the Team has been publishing their articles. I think it is sometimes difficult for Steve and others here to ignore how hard the Team has fought to avoid having their work replicated and evaluated by independent scientists, and their comments and reviews often carry an underlying thread of that resentment, and if you read anything at RC or Hansen’s diatribes, you will undoubtedly see that there is animosity and resentment on both sides. If you contrast that with the treatment given to authors like Loehle and others who have quickly archived their data, shared their methods freely, and been willing to discuss shortcomings and accept constructive criticism; you will find that the same resentment and animosity is not present FWIW.
@Susan–
Yes. There is a tribal feel here and at the other place.
I can say I enjoy CA because:
a) it has a more standard comments policy that permits conversation. Even if Steve sometimes drives the Zamboni through, you can read your own comments and responses in the standard order found at most blogs, and
b) CA has the rather unique “unthreaded” area.
On the other topic:
I’m not saying everyone will have read the all the same papers. There generally is a strong degree of overlap– as there must be for peer review to actually work. How can the system work if certain bodies of work don’t form some sort of foundation? In any case, the more specific point is, in certain circumstances, everyone will have read certain papers. 🙂
With luck, JEG will teach his students that R2 statistics should be calculated, and should be presented and that Steve M and Ross M were correct when they said so. If JEG doesn’t teach his students these things, a number will note JEG thinks R2 is important for Loehle, and draw their own conclusions.
They may also learn to imitate JEG and not to criticize powerful members of their field by posting public criticism even when that criticism is valid. And who’s to say that’s not a good life lesson for ambitious researchers? 🙂
Lucia, maybe this is a pet peeve, but I don’t look at it in terms of believing or not believing in AGW. It’s not a boolean logic question, it’s a fuzzy logic question. Even the IPCC makes an attempt at attribution. I think you meant that you believe that the anthropogenic contribution was significant. But “significant” leaves a lot of wiggle room.
Scientists have odious manners, except when you prop up their theory; then you can borrow money of them. (Mark Twain)
Okay, reading and/or skimming all that at once was rather painful.
Let me see if I can distill this somewhat.
1. Julien is in a position to critique Craig’s paper but not in a position to critique Mann’s previous work. Although interested in more open sharing of data and methods he’s not able to enforce any such thing on a long-standing field that he is new to. He accepts certain points and denies others (like all of us, subject to certain presuppositions) but doesn’t feel personally responsible for the work of others, pro or con.
2. Steve wants more open sharing of methods and data and some collaboration on creating multidisciplinary papers that involve multiple disciplines. He finds it perplexing that anyone would not hold everyone to the same standards.
3. Some people here, as in “the real world” are aligned more closely than others in their world-views. Neither “side” understands the basis and motivations and opinions on what drives the “other side’s statements”.
4. It doesn’t help anyone to call a paper, using a unique standpoint of using already peer-reviewed and finalized papers only, rather than original work, as being “psuedo-science”. Since quite clearly, we can’t even agree on what “science” actually is in this socio-political mess.
5. If the failure for the perceived lack of cross-validation of what’s perceived as a model is “woefully naive” or “downright unethical” and “lacks rigor in the crucial fulcrums of its reasoning” then on the other hand what does it say about combining peer-reviewed complete published papers? Are the papers complete (in which case they’ve been cross-validated and the point is meaningless) or incomplete (which doesn’t bode well upon the community that created them, nor their appropriateness as temperature reconstructions).
6. Complaining about CE, RE, and R^2 about Craig’s paper when not able to level similar complaints on previous work isn’t very helpful.
7. This process results in everyone talking past each other, when the effort would be better spent fixing things.
(And no JEG, you’re not in the hotseat for Mann, saying “I would have done it differently” is about as strong a thing to say as NAS saying that BCP shouldn’t be used for a temperature proxy. Which is what the discussion is about, IMO, degrees of meaning)
All I can say about this is that those willing to face the discussion and the heat about a contentious issue in an open forum that’s fairly free is to be congratulated. And I include everyone here in that. Nobody here is a troll and nobody here is to be censored rather than when over the top or totally off topic. It depends on if you can understand the other person’s viewpoint or not.
Can we start dealing with issues of substance, rather than arguing about the ratio of the radiative forcing of item x and how it fits into the whole?
BTW, SCO bought UNIX System Laboratory from Novell in 1995 (Novell had bought it in 1993 from AT&T) Currently, the trademark is registered to The Open Group.
Absolutely – everyone should have read certain papers in a given field. Look, to be honest, I responded because I felt there was a bit of a pile-on in response to Emile-Geay’s admittedly dismisive review. I was trying to point out what I saw as inquisitional behavior that went beyond what I felt was reasonable. My continued defense of my position was playing devil’s advocate — I am sure now that Emile-Geay has read MBH and the other paleoclimate papers and is quite familiar with this place, but at the time, I was not and may have been overly-generous. I try to be. I believe that openness leads to learning. If there are intellectual and emotional walls up and if there is the expectation of an incoming spear or bullet, defensiveness can close ears and preclude insight. I am trying to be open and polite so I can learn and not step on toes and ruing my chances of learning from people here what there is to learn.
I am here to see if I can learn about this “side”, what its goals are, what it sees as the issues and solutions. Having read some of the foundational materials, such as MBH, MM, Wegman, NAS, and other works, I am interested in watching what goes on here in order to understand and yes, to evaluate. Enough questions were raised in my mind about paleoclimate and its contribution to our understanding of the science of climate change and global warming that I felt a greater understanding was appropriate for me as a student of public policy.
As I understand it, no we can’t. Even though we’ve been indulging Susann, policy, AFAIK, is an off-limits subject. If it’s not, expect he(2x(hockey stick)) to break loose. You really want to start talking about cold fusion, and hydrogen, and all the other worms in that can?
There is no side here. We just want to know what’s true or not. Or at least I do.
Misdirection and obfuscation are not a way to impress me. If you don’t want to release some information, tell us why. You don’t, motives are questioned. Surprising?
When Dr. Wegman, whom as far as I remember, every statistician here considers the best in the field (and looking at his qualifications from a non-statistican standpoint like my own) finds problems with somebody’s statistical work…. Well. Especially work from somebody who has publically stated is not an expert statistician.
At one point, I didn’t understand this entire issue, but reading the report and the testimony has put it in perspective.
Actions speak louder than words.
Larry, substance meaning something from a common viewpoint rather than opining on matters such as which ice cream is better.
I agree though, at this time, no, we can’t.
Sadly enough.
@Susann– FWIW, I think you are seem to be doing a pretty good job. My statements aren’t meant as criticism, but just to shed light on some alternate views about the meaning of JEG’s demurrals based on my experiences. I’m assuming since your visits form a portion of your research you would not be taking what I say at face value, but likely asking others who have taken part in conferences whether their experience match my own. (Or you will if what I say is in any way relevant to your studies.)
Also– as to piling on, that happens at lots of blogs, particularly political ones. Visit a fat activist blog and suggest dieting is good for you, or a feminist blog and suggest women should defer to men or a anti-abortion blog and explain the positives of planned parenthood and watch the fireworks.
I think it will be interesting to read your thesis or papers based on it when you are finished. 🙂
Most open participation blogs and websites that I have visited that discuss issues that can be categorized contentious in the slightest have nearly the same amount of personality issues on average with some being worse than others but all having that character to a rather significant degree. Perhaps, knitting or gardening blogs or web sites avoid these issues, but even as an avid gardener I do not believe a gardening blog would be my cup of tea.
Blogs and internet discussions in general have been dominated by male participants and perhaps the testosterone levels in these discussions are taken more as a norm by males –and even older more docile ones such as I. I did, very briefly, participate in a web site discussion group for retired people and it did have more female participants than most, but its general tone too frequently devolved into complaints about personal problems and failure of government to attend to these problems. When I attempted to broach subjects such as SS and Medicare and how unfair it was to our children and grandchildren, I was never subjected to tribalism but rather long awkward pauses in the discussions.
Lucia, blogs — Go over to RC and ask why 100 ppmv is “IMPORTANT”
😀
Susann, I think that if you participate here and at RC in hopes of learning something about what drives public policy you may be disappointed and wasting precious time.
You, as I recall, have already given what I thought was an insightful view of what drives policy and that has more to do with political expediencies for politicians and less to do with the science as handled at RC and particularly at CA.
Perhaps you are unaware of Steve Ms comments likening his interests and motivations to solving puzzles. It is that element of his blog that I think attracts many people, including me. While Steve M remains firmly neutral on the political issues (and that I think attracts many people to his site as well) many of us have let our political standings be known and in my case I do because it affects the amount of certainty I need in the AGW issue. RC, in my view, is more policy driven from the top than Steve M is motivated by policy here at CA. CA attracts many who would oppose the RC policy and not because of Steve Ms policy stances, but because they are not RCs policy stances.
The policy battle, as affected by climate scientists and given the politicians motivations, has to be, in my mind, driven (given the political realities of today) by climate scientists abilities to point to climate tipping points and immediate consequences of AGW that voters will react to as something that will affect them in the near future and not something that might affect their children and grandchildren. I probably need not remind you that most develop nations in the world have huge unfunded liabilities for government health and retirement programs that the current generation has indicated they will ignore.
If politicians can formulate a policy that future generations pay for, like SS and Medicare, with no immediate detrimental consequences then I am confident that is how the policy will evolve. The policy contradiction is of course the need to show immediate crises while at the same time postponing the payments for the mitigations.
Without experiencing these detrimental consequences with hard evidence of AGW as the culprit I do not think that a scientific consensus or politician hand wringing will mean a thing.
[snip – any comment in which the word “socialist” occurs is 99.999% certain to be outside blog rules.]
You may not personally have taken a side, and may only be interested in “truth”, but that doesn’t mean what happens here isn’t used by those who have taken a position and are not interested in truth. That’s the risk we all take when we particpate here and elsewhere in such a politicized debate. Even those who truly are neutral can be used by those who are not.
Not surprising at all. I can only imagine the reason at this point: the reluctance could be because the request came from an outsider who was not part of the community and who might or might not be able to judge the contents. Mis-understanding the science could have negative consequences if unnecessary or unwarranted doubt is cast on the science, preventing it from moving forward. Perhaps there was concern that the data would be used in a politial manner. Perhaps because people involved knew the atrocious state of the data and didn’t want to be embarrased. Or a combination of those.
I really don’t know.
I read Wegman. I’m not by any means a statistician nor competent to judge him or his findings on statistics. However, I would think it might have been possible to find someone with expertise in both statistics and climate science, specifically paleoclimate, so that the particular way statistics are used by paleos would be better appreciated. I thought he over-reached, from what I understood. But I’m sure this will be unthreaded and I’m making a liar out of myself.
🙂
Steve: I think that there was considerable value in having a fresh face. His Reply to Questions is well worth reading.
I know I’m new here, but I happen to trust Susann to be doing what she says she is doing. Either that, or she deserves an Oscar for “Best blog performance by an undercover agent for the ‘other’ side.”
Ok… I admit it, I’m on the ‘other’ side, as I believe that the A in AGW is probably. Still, I think I get a sense of people who are straightforward. Like everyone, I can be fooled, but I’m not e particularly trusting; take my word for it, I’m not. 🙂
That said, some of this may belong on unthreaded (which I LOVE!)
Oh, I think he accurately appreciated the particular way statistics are used by paleos. He made fundamental methodological points. You don’t need to be a politician to point out the faulty statistics used by politicians. Just like you don’t need to be a paleoclimatologist to point out the faulty statistics used by paleoclimatologists.
Speaking as someone with statistical training and experience in application I can say that his statistical analysis was fine. He may have overreached in the cluster analysis, but that was just him trying to diagnose the source of the problem. The problem with the statistics remains regardless of the cause.
Lucia, I am not a mole from the “other side” – lol. You and I are probably in agreement on the A part, but I reserve the right to claim that I am undecided about the consequences.
I don’t think policy will be made on this blog or RC – not directly. Policy is made by politicians after weighing the costs that acting / not acting will have on $$ and votes. What happens here or “there” may have an impact on both and it is in that way that these blogs and others may or many not have an impact on policy.
/cynicism
@Susann–
I think blogs are like modern day coffee houses or gatherings over beer where issues are discussed. They do, however, have the additional the element of recording many tentative opinions (and unfortunately typos) forever, and sometimes (though not always) disseminating opinions and information rather widely; this can be both good and bad.
But no, no blog will drive policy on its own.
I reserve the right to not only claim I am undecided about consequences, I reserve the right to change my opinion on nearly everything. 🙂
I was reading at JEG’s web site in hopes of seeing something on how he or other climate scientists have rationalized, a prior, the use of teleconnections for explaining a specific temperature proxy. I read his papers linked by Steve Mosher on teleconnections and now his web site, but have yet to see the direct explanation for a temperature proxy and a rationale from him with regards to why it should be a valid use of teleconnections and not potentially confused with a spurious correlation.
http://thatstrangeweather.blogspot.com
On reading rather completely from JEGs site I would suggest any of you participating in this discussion who have not to do so. You can form your own opinions and judgments, but I view JEG as a scientist committed to climate policy and politics. Most of his observations appear to me to be reserved for what he views as the short comings of the reactions of skeptics and denialists to the climate science consensus. That well could explain the time he has taken to make those points here.
All that is neither here nor there to me if he can explain in detail some teleconnection validly applied to a specific temperature proxy. I have been sorely disappointed when previous climate scientists have visited here and appeared so edgy with what they have either assumed or observed was a general POV at CA that they never were able to answer any but the most rudimentary queries and then quickly leave.
225 Kenneth
I can’t find said links. I would appreciate it if you can point me in the right direction, or reproduce them here.
Thanks
Kenneth:
I agree: The tone and tenor of Julien’s site is very disappointing. Julien’s comments are somewhat odd: First cool and calm, then suddenly needlessly aggressive and antagonistic. I don’t get what he is really trying to do. Frankly, I am more interested in what he can add to Loehle’s paper.
Re #226
Start with http://scholar.google.com/scholar?q=Emile-Geay&hl=en&lr=&btnG=Search
One of JEG’s papers ( link ) addresses possible routes for solar influence on climate, with an emphasis on ENSO. It’s well-written. This may be of interest to those interested in that topic.
He’s grafted a radiant blade on a magnetic shaft; won’t stick.
=====================================
228
Thanks, John.
Re: #226
Abstract of JEGs 2007 dissertation while at Columbia:
http://digitalcommons.libraries.columbia.edu/dissertations/AAI3249076
That part is what I think most of us here understand about teleconnections and I judge that JEG misjudges in his apparent indictment of the general understanding of it here at CA. Thats the easy part. The more difficult task is in showing how teleconnections are utilized by a statistically valid method to explain a temperature proxy and detailing how teleconnected signals are extracted from the local signals.
Draft of a JEG et al. March 2007 paper while at Columbia on solar forcing during the Holcene through ENSO as a mediator:
Click to access Emile-Geay_etal2007inpress.pdf
I think that Steve Mosher misplaced these links that he posted on the thread “Somewthing New in the Loehle Network” so here they are. David Smith has also referenced this last link which given his interests certainly figures.
In his letter of October 15, 1992, to Professor Dr Robert Ehrlich, Editor, Journal for Mathematical Geology, Stanford’s Professor Dr A J Journel proclaimed, “The very reason for geostatistics or spatial statistics in general is the acceptance (a decision rather) that spatially distributed data should be considered a priori as dependent one to another, unless proven otherwise.” Professor Dr G Matheron taught Journal everything he knows. Matheron himself never applied Fisher’s F-test before his passing in 2000. Visit Wikipedia and look under Spatial dependence and Sampling variogram to find out what Fisher’s F-test is all about.