One of the examples of spurious regression mentioned in Phillips 1998, quoted by Eduardo Zorita at CPD and previously here by me, was taken from Hendry 1980, from an article entitled “Econometrics – Alchemy or Science?”. Hendry is a very eminent professor of economics and the article proved to be as interesting as its title.
Before I present Hendry’s example of spurious regression, here are some extended quotes from Hendry’s criticisms of econometrics, which are very reminiscent of views expressed here about multiproxy studies. The resemblance is not accidental as that is a framework that I approach the topic from. One of my regrets about the NAS panel was that they saw fit not to include a statistician qualified on these issues. Continue reading →
Jean S has sent the following longer contribution on Rutherford et al 2005. I always appreciate Jean S’s thoughtful comments (which is no secret to readers here). So enjoy. JeanS:
Now since the review of Burger and Cubash [BC06 here after] put Rutherford (2005) [R05] back on the table, there is an issue to which I’d like to draw readers’ attention. This issue went completely unnoticed by the AR3 of BC06 [I wonder if AR3 was a reviewer for R05 also, if so, what were his/her comments]. IMO, the issue is not fully noticed by BC, either. Namely that R05 does not use the RegEM algorithm by Schneider(2001), but a modified version of it. In fact, there are at least three completely ad-hoc modifications introduced in R05, I’ll discuss them below in some detail. Continue reading →
Mann, Wahl, Ammann, Ritson, Ramsdorff are all dining out on the fact that von Storch et al 2004 did not implement a detrending step in their implementation of MBH98. (Although given that Mann would not be "intimidated" into releasing his algorithm and cited Zorita et al 2003 to us as evidence that people could implement his algorithm, they should be embarrassed that people still could misinterpret their methodology. )
They also criticize the Echo-G model as though that had anything to do with the price of eggs. For the purposes of VZ04, the Echo-G model simply gave a set of gridcell results with a covariance structure.
Anyway, the pseudoproxies for Mann et al 2005 were archived. It was very easy to do the same calculations on them that I did last week on the VZ pseudoproxies. Guess what I found. Here’s a graph comparing detrended and non-detrended calibration for the 104 pseudoproxy network; the results for a 22-gridcell sample are very similar. So far I am unable to see that detrending makes a tinker’s damn in the results. (I’m not 100% sure of these calculations, but I’m pretty sure.)
NH Temperature – Black – NH average; blue – MBH; cyan – MBH detrended.
You’ll notice that Wahl et al. did not actually produce their own calculations. They merely hyper-ventilated in their article in Popular Science. In the Burger and Cubasch discussion, Mann using the sock-puppet Anonymous Referee #2 says that VZ should “come clean” about detrending. Even if true, what does that have anything to do with the review of Bürger and Cubasch? Is this what the Team has descended to? Again, I don’t see that the detrending/detrending matters a damn for the point being made, and, in no way, justifies the bullying.
I hope that you are following the lively discussion about Burger and Cubasch at Climates of the Past here , where Mann aka Anonymous Referee #2 is carrying on in a quite extraodinary way. I’ll probably try to weigh in over there at some point. The dialogue has exploded fairly quickly and I’ve collated some of the discussion in the post here.
First of all, I’m glad that the question of RE significance has been picked up. We raised the issue of spurious RE significance in our GRL article, but the only discussion of it so far has been Huybers 2005 and Wahl and Ammann (in their rejected GRL submission) both of which were critical and sought to restore the old benchmark of 0. Bürger and Cubasch, together with the Zorita review, are the second discussion to appreciate that there is a real issue with RE significance.
The first, interestingly, was the NAS Panel, which stated:
Regarding metrics used in the validation step in the reconstruction exercise, two issues have been raised (McIntyre and McKitrick 2003, 2005a,b). One is that the choice of “significance level” for the reduction of error (RE) validation statistic is not appropriate. The other is that different statistics, specifically the coefficient of efficiency (CE) and the squared correlation (r2), should have been used (the various validation statistics are discussed in Chapter 9). Some of these criticisms are more relevant than others, but taken together, they are an important aspect of a more general finding of this committee, which is that uncertainties of the published reconstructions have been underestimated. Methods for evaluation of uncertainties are discussed in Chapter 9.
I think that there are two separable issues in the Burger and Cubasch paper which you have to pay attention to –
1) the impact of a spurious regression on RE statistics;
2) the impact of calibration/verification period on RE statistics.
Continue reading →
RegEM has reared its ugly head again in Mann’s review of Burger and Cubasch. Continue reading →
Thompson et al said that "their [Thompson’s four Tibetan ice core] dO18 histories or proxy temperature records, are remarkably similar at lower frequencies". They are not just similar – they are "remarkably similar" with correlations as high as one milllll-ion. Continue reading →
There’s a short article “Model verification and documentaiton are needed” in Eos, June 20, 2006, by a geologist, I. Sasowsky, :calling for reviewers to ensure that computer methods are properly documented and archived as part of the review process. Sasowsky notes that prior studies have documented frequent “surprises” and “fundemental errors” in numerical modeling studies, even citing Naomi Oreskes (Oreskes and Belitz 2001) on this:
Trust, but verify”¢’¬?this is what editors ask for, and what readers expect, from reviewers of technical articles. As a reviewer, I am growing concerned with the level of trust requested by authors of submitted manuscripts, and the frequent lack of verifiable data and methods. Negative reports in the press [e.g., New York Times, 2005] attest to the worst-case outcomes of such shortcomings ….
Where scientific findings are based on computational analyses, documentation of computer model methods and analyses ought to be a required element of publication. The trust of the public in scientists and our methods depends upon this.
Ian Karucunas, the secretary for the NAS panel, kindly sent me the following reference to a June 20, 2006 announcement by the Research Councils U.K. (said to be equivalent to NSF):
The Research Councils UK Executive Group, the grouping of the eight chief executives of the UK Research Councils, has today published its updated position statement on access to research outputs (http://www.rcuk.ac.uk/access/). …
The paper reaffirms the Research Councils’ commitment to the guiding principles that publicly funded research must be made available and accessible for public examination as rapidly as practical …. and outputs must be preserved and remain accessible for future generations.
I guess no one sent the memo to Jones and Briffa.
There’s a new Lonnie Thompson article online at PNAS (thanks to Steve Bloom for reference). It has some "supporting data" – "supporting data to Thompson means only digital versions of the graphics, rather than detailed archives such as Majewski provided for the Everest ice core. Just to annoy anyone who was actually interested in the data and create a obstacle, Thompson archived this limited data as pdf rather than ascii. You can convert it to ascii, but it took me a couple of hours by the time that I’d sorted out different number of entries on each line. Continue reading →
Everybody seems to be so full of advice on how I should do things. OK, I have a question.
Ross and I are writing an op ed about the NAS Panel. It’s hard to convey to a lay audience what a comedown it is for the NAS Panel to say that it is merely the warmest year in 400 years. For example, a reporter at the NAS press conference called this statement "astounding", when it was a statement that would not have surprised Hubert Lamb or Reid Bryson or the pre-hockey stick paleoclimatologists. Continue reading →
Nature has an article on the top 50 science blogs taken from Technorati in which we do not occur. We have not registered at Technorati and am doing so now. The registration process requires that you do a post like this and insert the following in a message Technorati Profile
I’m not entirely sure where this blog will rank. If google scores are an indication, it would rank highly as we occur high on all kinds of google topics. The hit scores seem high, although there are apples and oranges in many comparisons. We tend to look at page hits – I think that we get a relatively high hit score per visit compared to other blogs, but who knows? On the other hand, we don’t rank highly in inbound links, which is a commonly used index.
Climate Audit 
Four Hunnnnnn-dred Years
Everybody seems to be so full of advice on how I should do things. OK, I have a question.
Ross and I are writing an op ed about the NAS Panel. It’s hard to convey to a lay audience what a comedown it is for the NAS Panel to say that it is merely the warmest year in 400 years. For example, a reporter at the NAS press conference called this statement "astounding", when it was a statement that would not have surprised Hubert Lamb or Reid Bryson or the pre-hockey stick paleoclimatologists. Continue reading →