Recently, Robert Way drew attention to Shi et al 2013 (online here), a multiproxy study cited in AR5, but not yet discussed at CA.
The paper by Shi et al (2013) is fairly convincing as to at least the last 1,000 years in the Northern Hemisphere. I am actually surprised that paper has not been discussed here since it aims at dealing with many of the criticisms of paleoclimate research. They use 45 annual proxies which are all greater than 1,000 years in length and all have a “demonstrated” temperature relationship based on the initial authors interpretations.
Robert correctly observed that Shi et al was well within the multiproxy specialization of Climate Audit and warranted coverage here. However, now that I’ve examined it, I can report that it is reliant on the same Graybill bristlecone chronologies that were used in Mann et al 1998-99. While critics of Climate Audit have taken exception to my labeling the dependence of paleoclimatologists on bristlecone chronologies as an “addiction”, until paleoclimatologists cease the repeated use of this problematic data in supposedly “independent” reconstructions, I think that the term remains justified.
While Robert reported that all these series had a “demonstrated” temperature relationship according to the initial authors’ interpretation, this is categorically untrue for Graybill’s bristlecone chronologies, where the original authors said that the bristlecone growth pulse was not due to temperature and sought an explanation in CO2 fertilization. (The preferred CA view is that the pulse is due to mechanical deformation arising from high incidence of strip barking in the 19th century, but that is a separate story.) As a matter of fact, by and large, the bristlecone chronologies failed even Mann’s pick-two test.
Shi et al also show a nodendro reconstruction. This has a much lesser semi-stick. This mainly uses a subset of Kaufman et al 2009 data. In a forthcoming post, I’ll show that even this weak result is questionable due to their use of contaminated data and upside-down data (not Tiljander, something different.)
Shi et al stated that they “started” with the 79 (Northern Hemisphere) proxies from the Mann et al 2008 proxy network, from which they discarded 60 series as “poor-quality data with coarse resolution”:
We started our proxy data collection using the large data set of climate records compiled by Mann et al. (2008), who collected 79 proxy records spanning 1000 yr to reconstruct NH temperatures. Of these series, 60 were not retained for this study, as they did not meet our very restrictive standards: e. g. some poor-quality data with a coarse resolution.
One very obvious quality control standard would be to implement the 2006 NAS panel recommendation that stripbark (bristlecone) chronologies be “avoided” in temperature reconstructions – a measure that surely ought to have recommended itself to the others, if only because so many prior studies have depended on Graybill bristlecones.
However, 10 of the 19 series carried forward from Mann et al 2008 were Graybill stripbark bristlecone chronologies (highlighted in yellow below):
Figure 1. Excerpt from Shi et al 2013 Table showing Graybill bristlecone chronologies.
Shi et al 2013 also appear to incorrectly believe that Graybill bristlecone chronologies have a “significant statistical relationship with the local instrumental temperature”:
Consequently, only 19 of these 79 proxy series were in common with both the present and the Mann et al. (2008) reconstructions. We also collected 26 other records, resulting in a total of 45 series used for our temperature reconstruction (Table 2). All references for the 45 proxies come from 21 studies (see Table 2). Here, every series was required to exceed a 90% confidence level with either one of the 2 closest instrumental temperature grid points over the calibration interval to ensure that it had a significant statistical relationship with the local instrumental temperature signal.
However, only a couple of these 19 series pass Mann’s pick-two test. It appears that Shi et al did not themselves verify this local correlation (since their Table 1 shows “ND” under the explained variance column) and relied on a presumed prior calculation in Mann et al 2008 establishing this supposed “significant statistical relationship”. However, Mann et al did not require such a relationship for inclusion in their overall network (which is a grab-bag), but only for inclusion in their CPS reconstruction. Mannian EIV methodology does not use this criterion – or even require that proxies be in the applicable hemisphere. Thus, Shi et al cannot assume that the inclusion of a proxy in the Mann et al 2008 network means that it has a “significant statistical relationship with the local instrumental temperature”: they seem to have misunderstood Mann et al 2008 procedures, something that is easy enough to do even for those for whom English is their first language.
But regardless of the reasoning of Shi et al for inclusion of these proxies in their network, the net result is that 10 of the 45 proxies in their network are Graybill bristlecone chronologies.
Many of their other proxies are familiar to the point of being sterotyped.
They use an older supersticked Yamal chronology (Briffa 2008). Plus Briffa chronologies from Taimyr and Tornetrask, a troika that is used in numerous multiproxy reconstructions and hardly new material.
Their running text is curiously silent (and even contradictory)on their use of Briffa chronologies (though they are listed in their Table 1). For example, they say of Tornetrask:
Briffa et al.’s (1992) and Grudd’s (2008) Fennoscandian treering sampling areas are the same (Torneträsk, Northern Sweden), but the Grudd (2008) record was updated from AD 501 to 2004 using new samples from 35 relatively young trees. This new Torneträsk treering maximum latewood density (MXD) record includes samples from a total of 100 trees and covers the period AD 441-2004. Therefore, we used the most up-to-date data from Grudd (2008).
Despite this, they used the Briffa’s Tornetrask chronology (under the alias “Fennoscandia”). This series was also used in Mann et al 2008.
They also use two Taimyr versions – one from Naurzbaev and one from Briffa. In their running text, they say:
Naurzbaev and Vaganov (2000) and Naurzbaev et al. (2002) both studied tree-rings in eastern Taimyr; the newer data by Naurzbaev et al. (2002) were used here
But, once again, they additionally used the Briffa et al 2008 Taimyr series (“Avam-Taimyr”). The Naurzbaev version that they used was also used in Mann et al 2008, where it was incorrectly attributed to Naurzbaev and Vaganov 2000. The Briffa 2000 Taymir version was also used in Mann et al 2008, though Shi et al can be forgiven for missing this: the “Tornetrask” series used in Mann et al 2008 was actually a composite of Briffa’s Tornetrask, Yamal and Taymir chronologies.
In fact, 27 of 45 (not 19 of 45) series are used in Mann et al 2008: 20 (not 19) ITRDB series, the three Briffa RCS chronologies, the Naurzbaev Taimyr version, the D’Arrigo Mongolia series, the Tan et al 2003 speleothem and Donard Lake varves.
Many of the series were also used in Kaufman et al 2009: all four ice core series; the five lake sediment series; the three Briffa tree ring chronologies and the D’Arrigo Gulf of Alaska tree ring chronology. The other series all appear to have been derived from Ljungqvist collections.
In tomorrow’s post, I’ll look at the lake sediment data, which we’ve been considering from time to time. Of particular interest are two series from Baffin Island. In connection with the recent interest in Miller et al on Baffin Island, I’ve re-examined the full Holocene information on Baffin Island lake sediments: there is convincing evidence that Shi et al used these series upside down (as have the many other users of these series) – an error that we previously observed in the PAGES2K use of Hvitavatn, Iceland sediments.
Shi et al 2013 applied several different methods to their dataset, all resulting in HS-shapes. At the end of the day, any reconstruction is a linear combination of the underlying proxies i.e. after much huffing and puffing, a vector of weights is calculated. One of the huge disadvantages of more “sophisticated” methods is that you can end up with series having negative weights i.e. being used upside down. On the other hand, you do want to downweight nearby series: e.g. multiple bristlecone series. However, this precaution doesn’t appear to have been observed in Shi et al.
But the main problem is the longstanding one: if you take a small subpopulation of hockeystick shaped bristlecones and mix them with a population of “proxies” that are indistinguishable from white noise/red noise and apply typical multiproxy recipes, you will get back a HS-shaped reconstruction. When you remove the bristlecones and the Briffa chronologies, is there still a HS reconstruction? I’ll start by examining the no-dendro chronology tomorrow.
Climate Audit 
131 Comments
Well, this shows what we all knew—bristlecone junkies will do anything for a fix …
w.
I should clarify that I simply repeated what was listed as criterion in the paper – I did not check to ensure the authors initial interpretations regarding temperature signal were retained (rightly or wrongly). During my read I was more interested in examining the differences between the reconstruction methodologies (EIV, CPS, PCA).
Steve: Robert, specialists in this field have placed far too much importance on complicated multivariate methods, in part, I suspect, because they tend to be outdoorsy, rather than mathematical, and do not have intuitive understanding of the underlying linear algebra, a topic that I’ve written on from time to time. If there is a consistent “signal” in the data, it emerges with simple weighted-averaging methods. More complicated methods run the risk of heavily weighting some series, flipping others. People who use “complicated” methods also tend to pay less attention to defects in the data e.g. Tingley and Huybers’ ludicrous inclusion of the contaminated portion of the Tiljander data, long after it was known to be contaminated. The larger issue here is more the continued passing-off of Graybill bristlecone data used in Mann et al 1998-99 as well as Mann et al 2008 in a supposedly “independent” study, as much as the (almost certain) incorrectness of their assertions of having carried out significance tests on local temperature.
Way:”I should clarify that I simply repeated what was listed as criterion in the paper – I did not check to ensure the authors initial interpretations regarding temperature signal were retained (rightly or wrongly).”
Don’t feel bad, Robert. The reviewers didn’t check either.
“I was more interested in examining the differences between the reconstruction methodologies (EIV, CPS, PCA)” and all of them are variants on correlation = causation. Deep science, that.
Steve, “they seem to have misunderstood Mann et al 2008 procedures, something that is easy enough to do even for those for whom English is their first language.” — too funny! 🙂 I’d forgotten how much I enjoyed Steve’s dissection of proxy temperature reconstructions. Glad you’re back at ’em, Steve.
Enjoyable and educational. We partly have Robert Way to thank.
Wrong in both directions. How do they manage that so often?
“Wrong in both directions. How do they manage that so often?”
It is a specialized area of paleo science called “Concurrent Suck & Blow”
Wind instrument players do have a technique called circular breathing which allows them to replenish their air supply while still putting out a stream of air sufficient to keep a note going.
This seems to be a positive, cool thing, as opposed to the consistent misuse of “proxies” by allowing them to be used upside down.
snip – any remarks to Way with even a hint of moralizing will be deleted
That isn’t really fair to say. Mr. Way is merely pointing out that he never claimed to have audited the paper’s data, rather, he assumed that things were done correctly and found the resulting analysis interesting. If Steve can show (as most here will expect given this post starting us off) that this paper is fatally flawed in data or methods, I’m sure that will change Way’s opinion of the paper and it’s results (or lack thereof).
Is the Shi & al paper and/or SI material available, except behind the paywall?
Steve: online here.
Baffin Island upside down! That should make a big splash.
A dog by any other name is still a dog!
I think Congressman Hank Johnson will be interested in this study.
Baffin Island upside down would also play havoc with melting ice and prehistoric moss results.
In a saner world Robert would have at least tried to discuss Steve’s commentary of Shi et al. In a mad world Robert would have learned by now to keep quiet as most the Great Climate Scientists do, having understood their funny HS business is built upon nothing at all.
I guess we’re somewhere in the middle here.
Steve: please – no more moralizing to or about Way.
Hi, Steve. I enjoy, as do thousands of others, reading your analyses of reconstruction studiea. It’s invigorating to readers.
And I enjoy seeing your snipping of personal comments aimed towards Way or anyone else. It’s good to see people focus on the issues raised and not on the personalities involved. Personal comments which correct the record are good. Negative comments about others, no matter how well deserved, leave a taint on the sender.
@Robert Way: “During my read I was more interested in examining the differences between the reconstruction methodologies (EIV, CPS, PCA)” Will you let us know the outcome, please, when you are through.
Might want to wait until Steve clears away any issues, there are already enough in this post to make the results in question until replication and correction at least of the data (to match the paper’s stated methods) can be done.
re: 2006 NAS panel recommendation on avoiding strip-bark chronologies
Salzer et al 2009 challenged this recommendation. How have Salzer’s own findings stood up since publication ?
There was some specific criticism at CA at the time (“Salzer et al 2009 – A First Look”) , but has there been anything further in the PRL ?
Salzer and co-authors continue to disagree with the NAS panel’s recommendation. From a more recent paper :
“Five millennia of paleotemperature from tree-rings in the Great Basin, USA”, Salzer et al 2013
Salzer et al 2013 is a waste of time if anybody wants to know if bristelcones are good for temp reconstructions, given that _that_ is the main assumption.
Salzer 2013 is open access at Springer http://link.springer.com/article/10.1007/s00382-013-1911-9
Steve: the new article discusses treeline in bristlecone areas – a topic discussed in early CA posts.
One of Salzer’s claims in just the abstract is disproved by simply visiting any Bristlecone stand…dead trees and debris above the current treeline contradict the claim that the current treeline is higher than in 4000 years.
Steve: they don’t say this and their SI shows otherwise. they say that the “rate” of increase in treeline is the fastest, not that it’s the highest.
Any particular insights that you enjoyed or found interesting in those differences? Particularly any methods (or pitfalls) relevant for discussion as Steve delves through Shi et al 2013?
I’m finding the paper has lead me to do more reading on the EIV approach which in turn has left me somewhat confused. Theoretically an approach like EIV if used on idealized meteorological station data makes some sense, I find the particularities of how one would use a data imputation method like this for proxy reconstruction interesting but needing some better description in the literature.
Steve: in linear algebra, the weights are the “dual” of the reconstruction. In my implementation of EIV (and MBH), I managed the linear algebra so that the weights were extracted concurrent with the reconstruction. One of many things that would be worth publishing. EIV is also a linear operation and results in weights. So you have to ask whether the weights “make sense”. If you know the orientation of the proxies (and you ought to if you know that they are proxies), then you should restrict weights so that they are all positive. EIV and PCA methods do not guarantee this as more factors are retained: in my opinion, this is a deterioration from simpler methods. IMO, 99% of the multivariate effort in this field is either pointless or bogus. Mostly because practitioners don’t understand the linear algebra in a thorough way and are easily led astray.
Wow!
It’s good to see Dr Way contributing here at CA. Beneficial all round!
“It’s good to see Dr Way contributing here at CA. Beneficial all round!”
Indeed – hopefully, he will stay around long enough to consider Steve’s criticisms of the paper in question and come to his own conclusions.
I think it only fair to warn him that should he chose to do so, and should it be the case that he finds the results, err, erroneous shall we say, and further that he mentions this in climate science circles, he will likely end up the same as Judith Curry has – labelled a “denier”, “mis-informer” etc.
Indeed, the nonbelievers are bad enough, but it’s the apostates that get hammered.
Seriously, though, continued use of stripbark chronologies bring to mind Marlborough’s quote ‘they came on in the same old way…’
Correction: that was Wellington.
At Waterloo and in reference to Napoleon’s use of closely packed columns of infantry advancing almost as a battering ram against infantry in line. OK against all but British infantry trained to fire at twice the rate of their European counterparts. (See Bonderchuk’s amazing movie, Waterloo (1971) ).
Substitute “McIntyre” for the British Infantry – largely Scots, anyway – and you complete your analogy.
Indeed, a sorry bit of brain fade on my part.
Y’all see the study now retracted by the Food and Chemical Toxicology Journal — The Seralini study published in September, 2012? …It purported to show genetically modified food as causing an increase in tumors.
http://news.yahoo.com/journal-withdraws-controversial-french-monsanto-gm-study-144512338–sector.html
Some relevant quotes…
“This retraction comes after a thorough and time-consuming analysis of the published article and the data it reports, along with an investigation into the peer-review behind the article,”
(an audit, perhaps?)
“Within weeks of its appearance in the peer-reviewed journal, more than 700 scientists had signed an online petition calling on Seralini to release all the data from his research.” — apparently no FOI request necessary.
One of the reasons for the retraction upon further examination was because of its small sample size not allowing its conclusions to be acceptably powerful/meaningful.
I’m constantly running into math outdoors. Is there shelter anywhere?
===========
I hear Bristlecone Pines have been sheltering certain people for quite a time…
Those trees sure do multiply.
I believe the trees multiply by adding the logs. Ugh, sorry, bad math joke …
McIntyre’s Razor has a Stihl edge.
==========
I will only say, that whenever some matrix operations are involved, you have to be very cautious in the interpretions.
It really makes me wonder why common averages apparently are altmodich.
Looking forward to the non-dendro analysis. Is there any way to rank the strengths of the various proxies that exist?
Fig. 4:
“To facilitate comparison, every series was
firstly expressed as anomalies from AD 1850- 1960,
and was then variance matched using the instrumental data from AD 1850- 1960”
Hey, I invented this before: http://www.youtube.com/watch?v=TTVRCXlqoT8 . I just used shorter anomaly/scale periods, shouldn’t matter.
Can you post up some code, or meta code, for that? Assuming that your box is a (moving) baseline, what are you matching it to? (obviously not instrumental data from the same interval as the box)
“what are you matching it to? ”
to mean 0 and std 0.1:
%recons: load juckesrecons
%recons: smooth juckesrecons (lowpass(in,1/40,2,2);)
N=length(recons)
tL=[1 30];
for ii=1:N-30
reconsm=nanmean(recons(tL(1):tL(2),:));
reconss=nanstd(recons(tL(1):tL(2),:));
reconsa=(recons-repmat(reconsm,N,1))./repmat(reconss,N,1)*0.1;
plot(time(:,1),reconsa)
aa=patch([time(tL(2),1) time(tL(2),1) time(tL(1),1) time(tL(1),1)],[-2 2 2 -2],’c’);
set(aa,’FaceAlpha’,0.9)
tL=tL+1
% fig to file
end
I dont understand…..Way exclaimed in comments on Cowtan and Way thread that the evidence was within Shi et al 2013 AND that this evidence was “convincing” for the last 1000 years.
Only after Steve Mc audited the paper does Way say he didnt read the whole thing and further that he doesnt understand the papers math…….
Am I wrong here?
Yes you are incorrect. I said I thought it was reasonably convincing because I liked the approach of using annual proxies, all covering the time period analyzed (e.g. no drop-off with time), with purportedly a relationship with local temperature* and testing the approach in the way they did (CPS, EIV, PCA). What I didn’t do was take their 45 individual proxies and look up every one of them for their peculiarities. I do not think it would be reasonable to be expected to do so with every paper one reads. I do not understand where you take that as meaning I did not read the whole paper.
Secondly, if you had read the paper you would note that there is not a clear explanation of the regularized EIV in the paper. To understand the peculiarities of this method you have to piece together from several other papers with the McShane and Wyner paper (and its large volume of responses and comments) being a good source for doing so.
Perhaps you’re letting your preconceptions about me impact the way you interpret the commentary.
Steve: Robert, I quite agree that there is no adequate discussion of RegEM/EIV in the literature. I ported the methodology to R and, in doing so, made some huge simplifications by understanding the linear algebra. It is unbelievably inefficient in Mann’s implementation, with the inefficiency demonstrating Mann’s apparent lack of understanding of the algebra. My blogpost here
https://climateaudit.org/2011/10/16/notes-on-regem/ contains notes on my implementation of the method, notes which lay out the linear algebra in a form not found elsewhere and which (IMHO) are superior to what you can find in the litchurchur. I have some other notes on the regularizing parameter that are quite interesting and which I haven’t put in a blogpost.
…and that’s why reviewers let dendro papers get published despite the papers’ extraordinary weaknesses…
Thank you for responding and I now understand your position.
I have never heard of you prior to the recent posts and I have no preconception of you. I am very impressed with your being on site and answering questions.
Actually, I have to second your compliment. Robert Way’s engagement here is a real positive. I would have liked an apology to Steve for past comments, but I’d rather have this kind of consistent engagement than an apology and a disappearance. Bravo Robert Way. As they say in AA “keep coming back!”
Do you still think that Shi et al is reasonably convincing, Robert?
Way, “I do not think it would be reasonable to be expected to do so with every paper one reads.”
Honest question here Mr Way. What value is a lay man to place in the Peer Review Literature if the peers are not critically reviewing the results to check their validity? It is understandable not to do so with every paper published, however, time and again papers and results critical to the status quo are demonstrated to be thoroughly flawed.
There appears to be an uncritical acceptance of papers because they were published, while the publishers themselves pass the responsibility of validating the results to the community. To say nothing of the difficulty in acquiring the data necessary to do so.
See, I agree with you. The problem is, Steve decided the would read the entirety of Mann’s paper and understand it. That was his original sin. Along the way, he ran into things that did not make sense.
As Ross explains:
Until I saw Steve’s work, my personal reaction to these papers had been one of intuitive dismissal … After all, I am used to seeing people who easily impress themselves with their ability to be fooled by spurious correlations.
However, the reaction by “climate scientists” to Steve’s discoveries told me there was something else going on. I am grateful that Steve has been checking the Team’s assertions in the intervening decade. He is a scientist in the true sense of the word.
RObert wrote: “What I didn’t do was take their 45 individual proxies and look up every one of them for their peculiarities. I do not think it would be reasonable to be expected to do so with every paper one reads.”
I cannot disagree more.
As someone who know the multiproxy papers very well, it’s the VERY FIRST thing that I do when I see a new multiproxy paper: examine the proxy network, see what’s old and what’s new. In the case of Shi et al 2013, for example, their use of 10 Graybill stripbark chronologies is very important in any assessment.
I also stratify the data to see which series are “active ingredients” in any HS-ness of the reconstruction, since too often results are dependent on a few series, despite protestations of authors otherwise. Thus, the contaminated Igaliku series in PAGES2K stood out immediately.
I know the proxies well enough that I am familiar with peculiarities of many of them, though some I’m still learning.
There are only a few “new” proxies in Shi et al. In my opinion, these are the most interesting aspects of any new study: do the new proxies shed any light on the field.
I have become profoundly uninterested in novel multivariate methods applied to datasets with bristlecones and Yamal in them. I am prepared to stipulate that they have a HS shape.
I am interested in whether proxies are consistent with one another in the same region.
Nor do I think that it is “unreasonable” for interested readers to know the peculiarities of the proxies. Thousands of hockey fans know the peculiarities of the candidates for Team Canada or of NBA players. I therefore think that paleoclimate scientists should know the peculiarities of dozens, if not hundreds, of proxies.
Maybe Robert’s reading of Shi et al was cursory, because he had forgotten about the issues with multiproxy papers that you and others have been exposing for many years. Perhaps Robert assumed that you had not discussed Shi et al, because you couldn’t find anything wrong with it. He is probably sorry he brought it up.
Yes and that is your niche and you’re welcome to find it interesting. You’re more than welcome to have your opinions on the matter and that is fine but if I read a paper and I see in a table of proxies that it lists a series of tree ring series as: ITRDB – ca606 (and other codes), then I feel it is not reasonable to expect that I have a responsibility to go check each individual series used from the ITRDB before commenting on whether a papers approach is interesting or not.
Were I auditing or reviewing a particular paper then perhaps that would hold more weight but as a person whose primary research subject is not paleoclimate it is odd to see that you have taken issue with this comment.
I think that the Shi paper presents many of things that have been asked for by people here and there should be some acknowledgement of this.
However, if there are issues with the proxy selection then those are legitimate concerns and I hope that they are not the major driver of the results.
As an example from your experiences – what would be a reconstruction paper that you feel would be an instance of reasonable statistical practice with reasonable proxies?
Steve: robert, I wasn’t trying to moralize about your individual reading of the paper if you are reading the papers as a non-expert in the field. However, I believe that experts in the field, including anyone who agreed to be a peer reviewer, should be familiar with the peculiarities of individual proxies. Thinking about this some more, I think that the authors of Shi et al should have disclosed clearly that 10 of 45 proxies were Graybill stripbark chronologies, that these chronologies have been used in many of the reconstructions to which the Shi reconstruction was being compared and that these chronologies had been sharply criticized, including a statement by the NAS panel that they should be “avoided” in reconstructions. Can I ask you a question on this: would your attitude to the Shi reconstruction have been affected if they had disclosed this to you? If it would have affected your attitude (and, as someone knowledgeable about reconstructions, it affects mine), then it should be mandatory disclosure. In my opinion, the absence of such disclosure is negligence on the part of reviewers, who should have asked for such disclosure. I agree with your observation that mere readers should not be expected to check the underlying papers. On your question, I’ve long stated that the problem in the field is proxy inconsistency and that this is not a problem that can be solved by fancier multivariate statistics. Only by detailed reconciliation, area by area.
Robert Way, you say you shouldn’t have to do things prior to “commenting on whether a papers approach is interesting or not.” You were quoted in this blog post saying the paper in question is “fairly convincing.” Do you think the same should be true for both?
I, personally, don’t think I’d be willing to say something is “fairly convincing” if I hadn’t applied any critical analysis to it. I certainly wouldn’t be willing to if the paper dealt with a contentious subject such as the hockey stick. I think that’s true of most people at this site, and I suspect it’s why you’re getting the reactions you’re getting.
Brandon,
You can be convinced by something that is wrong, as it is impossible to thoroughly investigate everything. The important thing is being open to modifying your opinion based on subsequent findings or arguments. If this paper, based on its text and no further investigation, appeared to meet and handle all prior types of criticism, then it might be convincing. If someone digs in and finds that the paper’s text does not, after all, handle prior critics at all, then the paper would cease to be convincing at this time.
As for whether or not Way should have investigated before opining that something was “convincing”, I would find little to no fault in that. It wasn’t as if Way was trumpeting the validity of this paper to the world without having dug into the nitty gritty, but rather, he commented (here on this blog) that it seemed to be valid and interesting. That’s fine, it’s just part of having a discussion. I would think somewhat differently if Way claimed to be a Paleo expert and recommended a paper without checking the data, but, that does not appear to be the case. Even if it was, it wouldn’t be a cardinal sin; it’d just be embarrassing.
CG, taking a paper solely at its word as resolving topics of dispute seems strange to me. I couldn’t do it. Even if I could though, this paper didn’t claim to do that. It didn’t come close to addressing most of the prior types of criticism. As such, I can’t see how one would come to believe it did.
One of the major criticisms of temperature reconstructions has always been the limited data containing a signal similar to the final result. As we see in this post, not only did the paper not address that issue, it reused much of the data which has been previously discussed. I don’t see how one can read the paper and think it handles such a major criticism, and I think that would be a bare minimum for finding the paper “fairly convincing.”
Re Way and McIntyre, above (and wishing that “Reply” were an option for every comment here): Feynman used to argue that every paper ought to present not only the evidence FOR its findings, but should also include every argument AGAINST its findings. This only makes sense if scientific truth is the object. And so here–seconding SM’s argument–Shi et al. needed, absolutely, to include “caveats” about their use of Graybill strip-bark chronologies and any reviewers needed to insist upon it. It is appalling, this frequent lack of scientific rigor in climatology publishing.
“Steve: However, I believe that experts in the field, including anyone who agreed to be a peer reviewer, should be familiar with the peculiarities of individual proxies. Thinking about this some more, I think that the authors of Shi et al should have disclosed clearly that 10 of 45 proxies were Graybill stripbark chronologies, that these chronologies have been used in many of the reconstructions to which the Shi reconstruction was being compared and that these chronologies had been sharply criticized, including a statement by the NAS panel that they should be “avoided” in reconstructions… Only by detailed reconciliation, area by area.”
I don’t disagree with any of the above statements much really. I think that the authors probably should have disclosed that this conflicted type of proxy was used and they should have provided justification for having chosen to use it in a similar vein to how they justified the use of other proxies. I am not an expert on this type of tree ring chronology but if I recall the NAS statement on that proxy contains slightly more caveats than simply do not ever use. If there is recent literature on the topic which is able to counter the NAS suggestion regarding this type of proxy then it should have been placed in that context within the data section of the paper. Lacking that sort of counterargument it is problematic and probably something that should have been required. However using loaded words such as reviewer “negligence” is beyond what I think is acceptable.
As for area by area – I agree with this approach and I believe it adds value to have actual experts on the given topic creating chronologies and producing regional reconstruction series. The alternative approach that we have seen has included downloading everything from a paleodatabase and running gridcell correlations etc… to build a reconstruction which I think of as being less convincing – even when done by statisticians.
Steve: I wasn’t trying to be overly emotive in using the term “negligence”. In this context, I merely meant “neglect” by reviewers – would this seem more apt to you? From my perspective, I think that reviewers typically spend too much time gatekeeping and not enough time in ensuring that the record of the article is complete in terms of disclosure and data availability.
Robnert wrote: “The alternative approach that we have seen has included downloading everything from a paleodatabase and running gridcell correlations etc… to build a reconstruction which I think of as being less convincing – even when done by statisticians.”
At the NAS panel workshop in 2006, Malcolm Hughes made an interesting comparison between two styles of reconstruction. (1) relying on multivariate methods to extract a “signal” from everything in a databank. Hughes called this the “Fritts method”, though more aptly the Mann method for most readers. Choosing small subsets from large collections is a lowbrow variation. (2) collecting data according to ex ante criteria and using simple averaging methods. Hughes called this the “Schweingruber method”.
The Schweingruber method is almost never used in current paleoclimate studies. Briffa et al 1998, 2001 were based on the Schweingruber dataset. This is the dataset with the famous decline, which Briffa failed to show in some important graphics, notably, in conjunction with Mann and Osborn, in IPCC TAR.
Mann’s reconstruction gave him fame and success, while Briffa’s occasioned zero interest and left Briffa near destitute for grant money. In my opinion, the contradiction between the Mann et al 1998 reconstruction and the Briffa et al 1998 reconstruction – both taken from large populations of tree ring sites – ought to have stimulated great interest among specialists to explain the discrepancy. Instead, in part because of Briffa’s decision to hide the decline, the divergence problem remained essentially unexplored for over a decade. Even today, the issues have mostly been papered over (through citation of the execrable article by Cook, normally a very capable scientist), rather than settled.
Re: Steve McIntyre (Dec 1 17:13), and Robert Way (Dec 1 22:06),
So… I would hope that both readers here as well as Robert and his associates would recognize that while statements have been made here about “agree” and “disagree”… in reality my sense is there’s vigorous agreement among the principals here on this topic:
* Any expert in the field, any reviewer, anyone with significant experience SHOULD be aware of the well-known datasets and SHOULD be checking such things in new papers, particularly because there is a long history of data reuse and data snooping.
* Authors and reviewers should (in best Feynman tradition) explicitly declare the well known caveats about data sets.
* Readers should not have to dig deep to discover those caveats and whether they apply.
To me, this is quite similar to declaring CI’s, or whether a medical study was done using proper double-blind methods, etc.
I don’t know about wrong but probably irrelevant.
This seems a fine example of science in action. Researcher reads paper and forms conclusions based on reasonable assumptions of compentence of original author(s), uses conclusions as part of further work and publishes, published work triggers further investigation and analysis impinging on researchers conclusions, researcher acknowledges issue exists and indicates further research.
That, a little inspiration and a whole lot of drudgery is how we advance!
A researcher who is prepared to attempt to explore ideas on a forum such as this deserves to be engaged on those terms. The discussion is about data and interpretation, not motive.
Selwyn said:
“This seems a fine example of science in action. Researcher reads paper and forms conclusions based on reasonable assumptions of competence of original author(s)…”
It seems a fine example of exactly what referees in the field have been doing.
Selwyn, apologies for not quoting properly, and appearing to disagree, when the wider scope of your comment showed the remedy Steve applied – which your comment was referring to.
In Bayesian theory, the correct weights used for each observation would be the inverse of the variance of the accuracy of the observation (also called the “precision”). If you use PCA or some other method to infer weights, and if the resulting weights cannot reasonably interpreted and scientifically verified to estimate precision, then the Bayesian will pronounce these weights to be sub-optimal. (BTW, an explicit data model showing you assumption as to the parametric form of the error process would also be helpful in evaluating the method.)
What happened to the comments? They look odd.
I realize this is a “weak” post in many respects. I’ve only dug in deep enough a few times to understand the math that Steve has been talking about, and mostly have taken for granted that you can’t tell much from small populations.
I’ll say simply, if I were to create a study, what is not to like about having Steve M. Review it? As I recall, A. Watts did this once, perhaps without enough time, and there were good comments from Steve.
Let me put it this way. I’m a skeptic. I’m skeptical because the world is a complex place, and it’s rare that anything with multitudinous variables, complex interactions, but one variable that does change over time dominates all others so much it’s “that variable,” unless the variable is something like “Smoke for 40 years, with hundreds of carcinogens,” and even then it’s not a certainty, as my 85 year old mother in law is find and kicking smoking her cigarettes.
To me, not including the good statistical analysis is foolish: it only adds doubt. Why not run the clean statistics? Who is afraid of the results?
‘Substitute “McIntyre” for the British Infantry’
Why does the invention of the Vickers machine gun suddenly spring to mind with its concomitant casualty rates? Not a pretty sight those battlefields with nary a tree stump, let alone any bark left on them.
Hmm, the site seems to have lost all its formatting. No header image, no CSS, just plain HTML styling it seems.
Anyone else seeing anything strange?
Re: Jeff Alberts (Dec 1 01:25),
Yup same here, Firefox, Windows 7
Re: Green Sand (Dec 1 05:12),
Same problems here in Chrome/Windows7 and Safari/iOs7; clearly a general problem.
My subscription to wordpress’ Custom Design feature had expired. I just renewed it. I think that this fixes the problem – let me know if it hasn’t.
Yep, it’s all good – have a great day!
Bummer, the mobile platform worked yesterday. Unfortunately that’s now fixed.
Good for me now.
Any chance you could put a clickable link to ‘mobile device viewer’ on the list on the left side? Someone once mentioned a (multistep)way to make the mobile view thing work.
On my BlackBerry, the pages show the leftside list and then the articles are all shown with the left ~20% of body text ruthlessly truncated. I can’t fix it with ‘pinch-pull’ dragging or any of the usual trix… Too many missing words to guess what they were…
TIA
AC
On my Android, I scroll to the very bottom and there is a link saying something like ‘View Full Site’ – that works.
The VIEW FULL SITE link at the bottom resolves nothing on BB Z10.
Just repeats issue of hard-truncation of leftmost 20% of text,
but not the left-sidea list of stuff starting at
Home
About
…
Subscribe to CA
Tip Jar
Maybe it is just a Canadian thing…
AC
EIV/TLS Regression – Why Use It?
http://statpad.wordpress.com/2010/12/19/eivtls-regression-why-use-it/
In one of the comments in the linked blog post, someone questions why proxies aren’t each first converted to a temperature, then treated in with methods similar to those used for, as an example, thermometer records. Dr. M. then discusses the difficulties converting proxies into a temparature. If it is difficult to do this for a given proxy, how does a mash-up of such proxies improve the situation?
Re: jim2 (Dec 1 16:22),
Jim2, let’s ask the CA user community a question …..
Is there a relatively concise process flow description out there on the Internet somewhere which contains a summarized step-by-step narrative of the Mannian process for creating a hockey-stick shaped paleoclimate temperature reconstruction?
A summarized process flow description would begin with collection of the proxy data, then describe the initial analysis of that data, then describe how data aggregation and data transformation processes are performed, then how data infilling is performed, and would then end up with a short description of the statistical derivation approach used for constructing the hockey stick shape from the combined/transformed data sets.
We would be particularly interested in knowing what kinds of data transformation processes occur along the way, including brief descriptions of how different classes of proxy source (trees, ice cores, sediments, etc.) are normalized into a common data analysis framework.
We would also like to know if any given set of data, either raw or transformed, which is passing through a particular process step does, or does not, need to have any kind of physical meaning attached to it — either before or after a given process step occurs.
I believe the code Mann used, or a reconstruction?, has already been dissected and found to be wanting. IIRC, basically, it filtered the proxies such that proxies that matched the instrumental record were retained, and these could be inverted to fit if necessary. The net result of this process is that the instrumental record will be at the end of the resulting chart, but all the data that comes before the instrumental period basically gets averaged. So the end result is the handle of the stick made up by the average of all selected proxies, followed by the instrumental record, which moved up to form the blade.
Re post By Bernie 1815 –
This is a classic example of corruption of fact by misguided belief – in climate change facts are often massaged to show what the presenter believes them to be and not what the facts actually show.
Below are the facts of the British regiments that fought at Waterloo , all those who fought were heroes no matter what part of the British Isles they came from.
British Regiments present at the battle:
Scots Regiments
Royal Scots Greys now the Royal Scots Dragoon Guards
3rd Foot Guards now the Scots Guards
1st Foot now the Royal Scots
42nd Highlanders now the Black Watch (the Royal Highland Regiment)
71st Highland Light Infantry now the Royal Highland Fusiliers
73rd Highlanders the Black Watch
79th Highlanders later the Queen’s Own Cameron Highlanders, then the Queen’s Own Highlanders and now the Highlanders
92nd Highlanders the Gordon Highlanders and now the Highlanders
British Regiments excluding the scots regiments
1st Life Guards now the Life Guards
2nd Life Guards now the Life Guards
Royal Horse Guards now the Blues and Royals
King’s Dragoon Guards now the Queen’s Dragoon Guards
Royal Dragoons now the Blues and Royals
6th Inniskilling Dragoons later the 5th Inniskilling Dragoon Guards and now the Royal Dragoon Guards
7th Hussars later the Queen’s Own Hussars and now the Queen’s Royal Hussars
10th Hussars later the Royal Hussars and now the King’s Royal Hussars
11th Hussars later the Royal Hussars and now the King’s Royal Hussars
12th Light Dragoons now the 9th/12th Lancers
13th Light Dragoons later the 13th/18th King’s Royal Hussars and now the Light Dragoons 15th Light Dragoons later the 15th/19th Hussars and now the Light Dragoons
16th Light Dragoons later the 16th/5th Lancers and now the Queen’s Royal Lancers
18th Light Dragoons later the 13th/18th King’s Royal Hussars and now the Light Dragoons
Royal Artillery
Royal Engineers
1st Foot Guards now the Grenadier Guards
2nd Coldstream Guards
4th King’s Own Regiment of Foot now the King’s Own Royal Border Regiment
14th Foot later the West Yorkshire Regiment and now the Prince of Wales’s Own Regiment of Yorkshire
23rd Royal Welch Fusiliers
27th Foot, the Inniskilling Fusiliers and now the Royal Irish Regiment
28th Foot later the Gloucestershire Regiment and now the Royal Gloucestershire, Berkshire and Wiltshire Regiment
30th Foot later the East Lancashire Regiment and now the Queen’s Lancashire Regiment
32nd Foot later the Duke of Cornwall’s Light Infantry and now the Light Infantry
33rd Foot the Duke of Wellington’s Regiment
40th Foot later the South Lancashire Regiment and now the Queen’s Lancashire Regiment
44th Foot later the Essex Regiment and now the Royal Anglian Regiment
51st Light Infantry later the King’s Own Yorkshire Light Infantry and now the Light Infantry
52nd Light Infantry later the Oxfordshire and Buckinghamshire Light Infantry and now the Royal Green Jackets
69th Foot later the Welsh Regiment and now the Royal Regiment of Wales
95th Rifles later the Rifle Brigade and now the Royal Green Jackets
largely Scots, anyway – I don’t think so!!!!!!!!!!!!!!!!!!!!!!!!!!
Another interesting post from Steve, who clearly knows what he’s talking about. That said,
I’ve seen little evidence that Steve is in anyway a great statistician. He may well be. However, the issues he raises are, sadly, rather fundamental. In other fields in which there is a greater emphasis on mathematical competence they would never arise.
There is a simple way for the paleo folk to neutralise Steve (and all the evidence suggests that this is exactly what they want to do). All they have to do is engage some competent outside statisticians to work on their papers. If I encountered several HS papers from independent groups which, as a matter of course, included professional statisticians (who are also prepared to openly discuss and defend the details of their findings) then I’d be convinced (and so, I suspect, would many other CA readers). The paleo community need to explain precisely why they haven’t done this since its such an obvious step. Sadly, the conclusion I draw is that the community is scared of the possibility/probability that their techniques simply aren’t fit for purpose and that their error bars will shoot up.
You mention that this “problem” is “fundamental” and should be easily dealt with by including statisticians!
So…..all of the bad models and the tuning could easily be made correct with a little better math?……sorry, that doesnt add up.
Why havent any of the “climate scientists” done this yet?
The paleo group AND all other “climate scientists” need to explain why—based on sloppy stats and baseless modeled date—my 5 year old child hears scare stories on her children’s shows about rising waters and exploding temps. I cannot even let her watch discovery channel or pbs without her getting the routine brainwashing. This effort to force their meme has found itself pushing through every and any form of media and is being touted as fact……..and the poor scientists, so maligned and so misunderstood, all they need to do is get a few statisticians to work the math around and POOF, it will all then be real.
Has it occured to you….1.that statiticians have already been asked and have already said that the maths are wrong?…..2.that the stats dept doesnt rely on scare mongering for their funding…..
Sorry, but did you read what Robin wrote?
Gaelan.
I get the impression you’ve replied to something I didn’t write.
I did not mention models; I referred explicity to the paleo work. Not did I write that a little better math would ameliorate the situation. Better maths may or may not lead to a credible hockey stick. I have no idea what the result would be in advance of such a study.
Regarding the point about statisticians declaring the work is wrong, can you please point out the body of work by professional statisticians pointing out that the approaches used in paleo reconstruction are flawed. I would very much like to see this. Occasional critiques of individual studies isn’t really enough. Evidence-based reasoning is important, especially in discussions such as these.
“”There is a simple way for the paleo folk to neutralise Steve (and all the evidence suggests that this is exactly what they want to do). All they have to do is engage some competent outside statisticians to work on their papers. If I encountered several HS papers from independent groups which, as a matter of course, included professional statisticians (who are also prepared to openly discuss and defend the details of their findings) then I’d be convinced””
From those words, I take this away: that you think the only thing these papers need is……..statisticians, and then you would be convinced. And that that particularly would end MrM’s challenges.
Sorry for reading what you wrote
Thanks.
My words summarised that I would be convinced by a thorough analysis by professionals who are able to master and defend the work, regardless of the nature of that result.
And here you have undercut the value of MrM’s work in your original first paragraph AND during the last decadehe has been the ONLY person providing independent analysis on the STATS, and somehow you still want to be convinced that the climate scare mongering community is right.
MrM’s work IS CONVINCING AND IS INDEPENDENT.
Describing Steve as someone who points out fundamental issues is hardly “undercutting” him.
Contrary to your assertion, its untrue to claim he is the only person doing independent studies. You should also give credit to Von Storch, Soon, Roman etc.
I doubt that anything I’ve written misrepresents the work which has gone on or disrespects those that have done it.
“”I’ve seen little evidence that Steve is in anyway a great statistician””
Save for his more than decades long battoe to show that the models are baseless…..upon the stats!
Clearly, you dont know what an underhand sleight it.
Yes, I’m sure if you ignore all of the positive things I’d written then it can be interpreted as a sleight.
As I mentioned earlier, you seem to be responding to things I didn’t write and looking for meanings which aren’t there.
Please, avoid cherry picking quotes – read them in context. Also, remember that others deserve credit – something Steve has never been reluctant to do.
I obviously took what you wrote differently than what you intended.
I apologize for that.
The conclusion I draw is that they tried that already, discovered for the third time that paleo reconstruction is a field before its infancy, and proceeded forward with their old,safe, incorrect ways rather than change employment
I certainly think there is a lot of fear. I rather suspect the main motivation now is the loss of face which would likely occur should they do things the right way.
We can’t see behind the curtains but I’d bet a hefty sum of money that the paleo folk are regarded with some embarrassment by other physical scientists. Steve + various CA contributors have made fools of them too many times.
Robin, have you looked at McShane and Wyner “A STATISTICAL ANALYSIS OF MULTIPLE TEMPERATURE PROXIES: ARE RECONSTRUCTIONS OF SURFACE TEMPERATURES OVER THE LAST 1000 YEARS RELIABLE?” (2011)? Does that paper meet the criteria you have in mind?
From their abstract:
“In this paper, we assess the reliability of such reconstructions and their statistical significance against various null models. We find that the proxies do not predict temperature significantly better than ran- dom series generated independently of temperature. Furthermore, var- ious model specifications that perform similarly at predicting tem- perature produce extremely different historical backcasts. Finally, the proxies seem unable to forecast the high levels of and sharp run-up in temperature in the 1990s either in-sample or from contiguous holdout blocks, thus casting doubt on their ability to predict such phenomena if in fact they occurred several hundred years ago.”
“I’ve seen little evidence that Steve is in anyway a great statistician. He may well be. However, the issues he raises are, sadly, rather fundamental. In other fields in which there is a greater emphasis on mathematical competence they would never arise”
A few things. First, statistics is not a branch of mathematics. Second, I am a statistician and I work with many of my kind. The best of the field are those whose develop strong intuition about the way real data behave. This intuition is gained by years of experience and is guided by mathematical competence. It doesn’t take sophisticated mathematical models, as John Tukey’s techniques for exploratory data analysis demonstrate. The host of this site doesn’t tout himself as a “great statistician” or even as just a “statistician” but from what I’ve seen his sense about data ranks with the best in my field.
“There is a simple way for the paleo folk to neutralise Steve (and all the evidence suggests that this is exactly what they want to do). All they have to do is engage some competent outside statisticians to work on their papers.”
The problem is that a competent statistician likely would raise the same issues pointed out on Climate Audit. Or is it a team player that you’re looking for? We call those “rubber stamps”.
Steve: everything that I do is technically pretty much within the scope of a university undergraduate in mathematics of my day. The corpus of a “great statistician” is completely different and well outside my accomplishments, such as they are. I think of my analyses as data analysis, rather than “statistics” as it is now done in academics. I do not regard anything that I do as showing anything more than a degree of talent. I think that I have a particular talent for seeing patterns and oddities in large data sets but this is practical rather than theoretic.
Bob
“The problem is that a competent statistician likely would raise the same issues pointed out on Climate Audit. Or is it a team player that you’re looking for? We call those “rubber stamps”.
I’m somewhat surprised by your reaction. Why do you think I’m looking for a rubber stamp ? Can you please explain with reference to what I’ve written (as opposed to anything I haven’t written).
Robin, I didn’t say that you were looking for a rubber stamp, but I wanted to show the dilemma raised by your suggestion. Putting a competent statistician on a team where statistical competence has been wanting presents a real danger of independent thinking on the part of the statistician.
Re: Robin (Dec 2 07:25),
You’re correct that many of the problems are fundamental; that’s part of what is so galling about the obvious issues. Perhaps you’ve missed the long-running request here for exactly what you propose: more involvement of qualified statisticians in the work.
In that sense, you’re also correct that in some ways the work done here usually doesn’t require world-class stats expertise.
However, if you take the time to read the CA history, you will find three realities that perhaps speak to your question, and explain why it’s essentially impossible to “neutralize” what is said here:
1) Quite a few high level statisticians (as well as experienced professionals in other related fields) contribute here. Steve is not walking alone.
2) As has been noted by others, one mark of experience is an innate familiarity with the data sets. In the past, Steve sometimes took on a fun exercise: by examining only the output of a reconstruction, he was able to largely predict which data sets were used as input. That’s one of many metrics that garner respect.
3) Steve is very careful. It’s quite difficult to catch him in a significant error.
Mr Pete
My first comment, which seems to enraged so many to the extent that they didn’t bother to read past the first few words, was pointing out that the mainstream paleo folk are most likely avoiding external experts for fear of the results.
I shan’t take up any more space on Steve’s blog though I would be surprised if he would have any issues with anything I’ve written.
The whiff of tribalism here is a little unfortunate.
Steve: nor do I particularly disagree. During the NAS workshop in 2006, I discussed the consult-a-statistician recommendation with von Storch and Zorita. Von Storch, who knows a lot of statistics, said that such consultations had mostly been unproductive in the past since academic statisticians typically wanted to apply methods that were inappropriate to the data. Actually econometricians have more intuitive feel for problems of autocorrelation, heteroskedasticity and spurious regression than a typical academic statistician. This is one of Ross McKitrick’s strengths. But problems like data snooping are very lowbrow and not something that an academic statistician can really get interested in.
Robin -your words could and perhaps should be interpreted as a stiletto hit against Mannianism. I’m sure if the Great Scientist could’ve found a statistician to back up his pretences, we would’ve been told left right and center.
As the controversy is 15 years old, the chance of that happening is negligible.
Re: Robin (Dec 2 13:29),
By my count exactly one person responded a bit upset at how you said it.
If one person makes a tribe, then we all have some deeper issues 😀
Steve – thanks for responding to my post.
Although such an endeavor may be doomed by gatekeepers, a general review paper pointing out how the body of paleo reconstructions uses biased methods and inappropriate datasets would be extremely useful. A lot of people would love to know how many reconstructions in the literature are left standing after basic common quality criteria are applied to the methodology. As you certainly know, there is a lot of expertise and familiarity possessed by you + various CA posters.
Such a paper would continue to promote the usefulness of non-affiliated scientists and the role of blogs. It would also secretly be devoured by a generation of graduate students in the field. Most love to see egg on their supervisors’ faces (regardless of what they may claim on forums). This is how one affects change in science. There are too many vested interests (academics, funding agencies, universities) to expect change there (note to Lew – this *isn’t* a conspiracy). They have invested a lot of reputation and finance in the paleo debacle. Occasionally, strong leadership can turn things around but the “leaders” of the field seem intent on throwing abuse rather than engaging in the substance of the argument. To me this behaviour is one of the most depressing aspects of the whole thing.
On another note, I also tend to regard the stuff you do as being data analysis rather than statistics, as I do in my job as a subatomic physicist. When we found the Higgs going to two photons it was with a simple weighted average. Nobody shouted for a more sophisticated treatment – this comes after the discovery.
Robin wrote:
“a general review paper pointing out how the body of paleo reconstructions uses biased methods and inappropriate datasets would be extremely useful.”
Try this one:
Click to access WegmanReport.pdf
Robin, in addition to gatekeeping concerns, such a paper might be doomed simply because of the amount of material it’d require. I don’t think you could hope to discuss all the reconstructions. That’s why I’ve previously suggested it’d be nice to have a website which does exactly that. The site could have a page for each reconstruction, and that page would discuss the reconstruction’s data,* methodology and why those led to the paper’s results. With enough interest, it could even provide the data used in each paper, as well as any code which is available.
That’d be a massive undertaking though. I don’t know who’d be willing to do it. I’ve spent a great deal of time following the debates over temperature reconstructions since before this site existed, but I’m sure I don’t even know half of what would be needed for a site like that.
I’m also pretty sure you wouldn’t be able to get funding for it. That amazes me because a resource like that would be a far more useful contribution than yet another paper using overly-complicated methods on much of the same data as before.
*Ideally, data would also have pages of its own. That’d allow people to easily check what concerns have been raised about what data.
It occurs to me even if authors didn’t provide code for their reconstructions, a site like that could still provide code for replications of those reconstructions. That means the site could provide packages for reconstructions that include data and turnkey code for those reconstructions.
If global warming is the serious problem so many people portray it as, I think sites like that should already exist. In theory, a site like that could exist for the global warming issue as a whole. It probably wouldn’t even cost as much as one GCM, and I think it’d provide a lot more value.
Brandon
There is a role for both academic papers and web info. A review fits best in journals IMO.
The undertaking would be fairly large. However, when writing review papers (I’ve written a few) the trick is to define and limit the remit of the paper appropriately. One remit could be, eg, to consider the top y papers (in terms of citation yields) for different regions and show whether or not they match basic methodological requirements. Such requirements could be, eg, reproducibility, methodological bias (eg does noise also produce the shape of the published result), statistical precision (does the conclusion rely on one tree… ) etc. Much of the info has already been published on CA in one form or other.
Its certainly easy to suggest these things and not so trivial to do them. However, I don’t see it as a huge task should somebody competent and knowledge have the time and interest to do it.
The gain would be, for the first time, an assessment of the body of work against the same common set of statistical standards. There is a famous result in medical sciences showing how the efficacy of drug X depends on the quality of a study – the true efficacy being found in the asymptotic limit of high quality. It would be very interesting to see if any such results emerge in this case. This is very much a “statistics” paper and I could even see it evading the gatekeepers and being published in a stats journal.
I have never seen a critical review such as you describe in any Statistics journal. Most reviews of a topic summarize the available material without serious criticism of failed methodologies which are simply omitted from the conversation.
On the other hand, the climate paleo community seems to have attempted an organized project whose aim was to evaluate the characteristics of different techniques for paleo reconstructions. The Paleoclimate Reconstruction Challenge appeared in January, 2008. After indicating that there were indeed some differences in the various reconstructions, the announcement goes on to state:
Data sets were provided for the project (part of which was used for a post here on the possible effects of correlation screening of proxies) and I expected to see a good scientific evaluation of the current reconstruction methodologies. Five years later, nothing has appeared.
There is currently a web page at NCDC which states:
and a quick google search brought up a blog site offering an explanation of possible reasons for the lack of results:
I have my doubts that the challenge will happen in the near future.
Robin, I’ve never seen a paper like that, and I can’t imagine how it’d work. I wouldn’t be opposed to anyone trying though. I just have little hope it’d succeed. I don’t see how one would compress the material enough without people deciding it was done inappropriately.
As an aside, I’m going to look into how much trouble my idea would be. I think there are about three dozen papers which would need to be covered. That’s not too bad. It might even be viable as a hobby if one was willing to post it as a work in progress.
Thanks Roman for the info.
Brandon – I’ve written more complex reviews (in another field, subatomic physics) than that proposed here. Its not a huge undertaking, so long as someone (or a group) has the energy, interest and expertise to do this.
In fact, in a better functioning field there would already be several such reviews in print.
Re Robin:
***I’ve seen little evidence that Steve is in anyway a great statistician. ***
What you should have noticed is that he stands above most of the people in the climate weblogs who “claim” to know what they are doing. He does this for a hobby not daily living.
***There is a simple way for the paleo folk to neutralise Steve (and all the evidence suggests that this is exactly what they want to do). All they have to do is engage some competent outside statisticians to work on their papers.***
You are dreaming. This has been suggested by Steve. This would cause a rejection of many of the papers reviewed here and also used by IPCC. You will also notice some of the warmist weblogs continue to justify bad papers including “independent” studies showing the “hockey stick”. Something like the 97 percent consensus paper would have died long ago.
Robin, your recommendation of working with competent external statisticians has been made repeatedly to the climate science community, for example by the Wegman report in 2006 and the Oxburgh report in 2010. These recommendations seem to have been ignored.
Climate etc has this link to an article about Climategate and Steve’s role. T think it’s brilliant.
http://michaelkelly.artofeurope.com/cru.htm
What’s truly remarkable is that Michael Kelly wrote that four years ago, and the ignorant juggernaut rolls on.
===================
Rolls on in the same old way, I might add.
=================
Speaking of epic battles over trees I wonder if it’s still there-
http://en.wikipedia.org/wiki/Battle_of_Lone_Pine
This is a perfect example of science as a social -or perhaps even personal- construct. For Robert and presumably many professionals in the field, Shi et al is just another contribution of possible validity and importance. Things are moving in the paleo world etc.
For me and probably many commenters here, the mere mention of Graybill is the signal of a bull***t extravaganza, given its usual accompaniment of reticence, forgetfulness and general inability by the authors to defend their ground from age-old questions. The whole paleo field is irredemeably poisoned and can’t provide any useful information etc.
I surmise that an outsider would not associate science with reticence.
I can’t add much that hasn’t already been said so thanks Steve for continuing to do the hard archeological work.
I suppose that I could write that I’m surprised much of the variance loss concerns which have been published at this point haven’t eliminated hard correlation sorting in paleo-literature. Unfortunately, I’m not that surprised though.
I suspect Robert Way made a quick judgment/comment on this paper because it supported his POV on the matter. This is a very human trait one can see on all sides of the AGW issue. I think such exercises should make it evident that a public/blog figure commenting, in what might appear to some to be in an authoritative manner, on a particular part of the science related to AGW may well not have the background knowledge to do so. I would hope by now those reasonably people interested in these issue know that they should read the papers and make their own judgments.
I would also note that while it is of interest to many to see and make analyses of the methodologies, calculations and interpretations used by those authors doing temperature reconstructions, in almost all cases of which I am aware those authors continue to make the same basic mistake. The temperature proxies cannot be selected after the fact of measuring the correlations of the proxy responses to the instrumental record. Most proxies that respond to temperature also respond to other variables and have temperature signals that must be found in these relatively high levels of noise. If one can assume the noise is random then the only reasonable method of finding the temperature signal is to find a reasonable and physically based method of selecting proxies prior to any measurement and then using all proxy data. If the noise is not random then not even this method of averaging out the noise will work. All these relationships need to be determined prior to doing what could be considered a valid reconstruction. Unfortunately those authors doing temperature reconstructions do not start from the beginning and thus the shortcut they take of selecting only those proxies that show the recent warming in their overall response has to bias the final results.
I just reread RomanM’s post from some time ago at CA from the link in his post above where he gives mathematical rigor to what I was attempting to say above. His analysis and the example of the fishing net with holes in it makes clear the very basic problem involved in almost all temperature reconstructions.
Why this problem has not been addressed by climate science is a bit puzzling? It is as if it is a conceptual blind spot. I know there are many hard scientists, used to doing preliminary testing before making a confirming test, who evidently do not understand that confirming tests are not always possible in the softer sciences.
There is a “rerun” of Calvin&Hobbes comics on the web. Somehow the last panel of “today’s” comic might amuse one or the other here:
http://www.gocomics.com/calvinandhobbes/2013/12/03/
Direct link to the larger image:
http://assets.amuniversal.com/93c18e40172a01312ddf001dd8b71c47?width=900
There’s plenty of reason to anticipate less-than-desirable results from almost any study… much more humility needed.
Note sure if CA has already had a discussion of Why Most Published Research Findings Are False. Interestingly, this is one of the most-cited papers out there, and the author, John Ioannidis, is well accepted among peers.
Here’s a popularized view.
I’ll also note that Prof Ioannidis is the one who is initiating a project to investigate issues of invalid outcomes in science research and identify potential process changes. Watch for the “Meta-Research Innovation Center at Stanford (METRICS)…”
The problem is far more pervasive than I ever imagined. Another paper showed that of the 49 most highly cited medical research results in the most prestigious medical journals, 34 had been later retested, and 14 of those retests either contradicted or had significantly smaller effectiveness than first reported. A 41% error rate among the “best” papers.
For every over the top temperature response for proxies like bristlecones there are no doubt under the bottom responses – think divergence problems. These over/under responses may well “average out” if one were to use all the available data and not select after the fact like is the case in almost all temperature reconstructions.
The trouble is that mainstream scientists don’t seem to care whether these papers are correct or not and most don’t bother to read stuff outside their own field. Any criticism is seen as just speading doubt about an obvious truth.
In my own field I have wasted countless hours trying to replicate papers only to discover that the paper a) just doesn’t work, b) works only for very limited cases, c) is worse than knowledge already out there or d) is a pure academic exercise with no practical utility. I get maybe one paper out of fifty to be useful – hence I just skim most. I’m guessing but maybe that’s the case for most folk.
I remember Michael Crichton mentioning the effect of criticising a newspaper for getting something utterly wrong about your own field but being still prepared to believe the rest of the newspaper on subjects you know less about. It changed the way I read the newspapers!
“When you remove the bristlecones and the Briffa chronologies, is there still a HS reconstruction?”
Do you mean that after almost a decade you are finally going to get around to answering this question? I assume we can look for more on Salzer in…2019?
Steve: the HS in earlier multiproxy reconstructions depended on problematic proxies. Shi et al 2013 is a relatively new paper and was not available 10 uears ago, so I can hardly be expected to comment on it before it was published. Because Shi et al used bristlecones and Briffa, it is not “independent” of the earlier reconstructions. It has a few proxies that I haven’t previously parsed; I’ve taken a preliminary look at them and they don’t appear to have a HS. But I want to doublecheck this before posting on it.
Okay, but when you do, be sure to define “hockey stick shape” clearly–are you talking blade or shaft or what? Orientation? Magnitude? The definition seems to change quite a bit.
Steve: for blog posts, I presume that readers are familiar with the concept. We used the following definition in McIntyre and McKitrick 2005 (GRL):
I don’t think that there’s anything magical about this definition, but it was serviceable for the purposes of that paper. I think that my usage has been consistent over time, though (and I presume that this is what you have in mind) the shape of the Mann et al 2008 reconstruction and some other recent reconstructions are not as HS-shaped as Mann et al 1999 and is more like Moberg 2005. Their main unifying feature is that the modern portion tends to be a fraction higher than the medieval portion, though the differences are generally razor-thin.
Wait, what? Since when does a hockey stick shape need to be defined? Can’t we just look at a hockey stick and see?
I get some people may label reconstructions that are nowhere close to resembling hockey sticks, hockey sticks, but why should people misusing words require us define those words clearly?
Shouldn’t Boris be taking this issue up with the people misusing the phrase?
Mann seems to have a bit of trouble with this. He thinks his 2008 publication confirms his earlier 1998 paper.
It’s the only field I know where two completely dissimilar results can be seen as confirmation of each other.
Could I suggest a more functional definition of a “hockey stick” curve as it applies to climate science. I would suggest that a climate science hockey stick temperature curve is a curve which shows a high degree of correlation with the atmospheric CO2 concentration. This definition relates to the ability to create a transfer function between them. a high degree of correlation would indicate a simple transfer funciton with little influence from other factors. Hence a diminished role for “natural variation”.
So “hockey stickedness” would be the correlation between normalized CO2 and temperature curves.
This measure would seem to reflect the reason why “hockey stick” curves became a subject of interest in the field.
“Mann seems to have a bit of trouble with this. He thinks his 2008 publication confirms his earlier 1998 paper.
It’s the only field I know where two completely dissimilar results can be seen as confirmation of each other.”
Exactly, Carrick, I thought we were going to get a concession speech from Mann after Mann(08) – even though I totally disagree with the selection of proxies after the fact approach in that paper.
One Trackback
[…] https://climateaudit.org/2013/11/28/bristlecone-addiction-in-shi-et-al-2013/#more-18641 […]