M&M Feedback by Anthony Lupo

Feb 8, 2005 – 1:08 PM

We received this comment on the research of McIntyre and McKitrick from Dr. Anthony Lupo, Professor of Atmospheric Science, University of Missouri-Columbia:

I will confess that I was not aware of the details of Steve McIntyre and Ross McKitrick’s critique of the "hockey stick" but after a cursory reading of the enclosed materials it seems that the critics have valid points.

I’ve been skeptical of the "hockey stick" for a long time simply on the grounds that there is too much evidence that climate has been more changeable than the "hockey stick" would indicate, even if much of the evidence is from a more limited region of the globe. Also, having taken part in the IPCC review process for the 2nd and 3rd assessments, I was continually frustrated with drafts that had: [will include text later] [will insert figure here] riddled throughout them. Thus, I’m not surprised that some may have made errors in their science and then, for whatever reason fail to provide their methods.

Again, I’m not an expert in tree ring studies, but Steve and Ross’s work to me makes good points. I’m happy to see work like theirs get published.

By Stephen McIntyre | Posted in News and Commentary | Comments Off on M&M Feedback by Anthony Lupo

Moderation

Feb 7, 2005 – 9:26 PM

I’m just learning how to do this, with the admirable help of John A. Comments are lightly moderated – which is really only aimed at spam. I just learned how to put the comments through. I’ll try to have comments cleared at least twice a day.

By Stephen McIntyre | Posted in General | Comments (1)

Spot the Hockey Stick! #2 The Kyoto Flames Cheerleaders

Feb 7, 2005 – 7:38 PM

I defer to Roy Spencer on the subject of why the Hockey Stick was taken so seriously and so completely (my emphasis in bold):

The hockey stick, along with the “warmest in 1,000 years” argument, has become a central theme of debates over the Kyoto Protocol, a treaty to limit emissions of greenhouse gases, in governments around the world. The question begging to be answered is: Why did the IPCC so quickly and uncritically accept the Mann et al hockey stick analysis when it first appeared? I cannot help but conclude that it’s because they wanted to believe it.

Alas, the Hockey Stick affair is a clear warning about the value of skepticism in science, especially of results which confirm your private prejudices.

Be that as it may, here is a selection of cheerleaders for the “Kyoto Flames” Hockey Team, in no particular order of gullibility:

The Heat is Online or was that “hype”?
“Outstanding Student Papers on past temperatures” [pdf] from University Corporation of Atmospheric Research
National Center for Atmospheric Research
Australian Greenhouse Office[pdf] Chapter 2, page 4

References but no picture of the Hockey Stick, include:

Greenpeace
Union of Concerned Scientists

By John A | Posted in Spot the Hockey Stick! | Comments Off on Spot the Hockey Stick! #2 The Kyoto Flames Cheerleaders

Was Preisendorfer’s Rule N Used in MBH98 Tree Ring Networks?

Feb 7, 2005 – 9:33 AM

Mann et al. have recently argued that they can salvage MBH98-type results, even with correct PC calculations by also using “the standard selection rule (Preisendorfer’s Rule N) used by MBH98”. They say that this method permits them to retain 5 PCs in the North American network. The reason why this matters to them is that the bristlecones are represented in the PC4 in correct PC calculations and the expanded roster would permit them to imprint the NH temperature reconstruction even from the PC4 position.

We have discussed elsewhere many issues regarding the robustness and statistical significance of this calculation, arguing that the bristlecones are not a valid proxy, that the MBH98 reconstruction is not robust to the presence/absence of the bristlecones and the MBH98 reconstruction has no statistical significance in its early portion. Here I consider the narrow issue of whether the PC selection method illustrated in the Nov. 22, 2004 diagram was actually “used by MBH98” for tree ring networks.

I have been able to closely replicate the Nov. 22, 2004 diagram published at realclimate.org – which is now said to demonstate the PC selection method used in MBH98 (although this method was not reported in MBH98). I have tested the 19 network/calculation step combinations used in MBH98 to verify the archived PC selections at the Corrigendum SI against this diagram.

In 18 of 19 cases, the selections from the Preisendorfer-type calculation are inconsistent with the the Corrigendum SI archive. In some cases, more PCs are selected; in some cases, fewer. In 3 calculations, different selections are taken from the same network in different calculation steps – a result inconsistent with the stated policy.

This is strong evidence that the method of the Nov. 22, 2004 diagram, as illustrated and without still undisclosed modifications, was not used to determine the number of retained PCs in MBH98. We believe that climate scientists, who have relied on MBH98, should ask Nature and/or the U.S. National Science Foundation to require the disclosure of all source code used in MBH98, including the selection of PC series.

MORE

Mann et al. have recently argued that they can salvage MBH98-type results using correct PC calculations under “the standard selection rule (Preisendorfer’s Rule N) used by MBH98”. http://www.realclimate.org/index.php?p=8  They say that this method permits them to retain 5 PCs in the North American network. Since the bristlecones are in the PC4, this expanded roster still permits them to imprint the NH temperature reconstruction. We have discussed elsewhere many issues regarding the robustness and statistical significance of this calculation. Here I consider the narrow issue of whether this method was actually “used by MBH98” for tree ring networks. I have been able to closely replicate the diagram published at realclimate.org on Nov. 22, 2004, said to be an example of the selection method used in MBH98. I have tested the 19 network/calculation step combinations used in MBH98 and, in 18 of 19 cases, the selections from the Preisendorfer-type calculation are inconsistent with the reported selections at the Corrigendum SI. In some cases, the results are higher; in some cases, lower. In three calculations, different selections are taken from the same network in different calculation steps – a result inconsistent with the stated policy. We remain puzzled why Mann et al. continue to refuse to provide source code for MBH98 calculations and why climate scientists do not expect them to do so.

Statements in MBH98

First, there is no mention in MBH98 or the MBH98 SI that Preisendorfer’s Rule N was used to determine the number of retained PC series for tree ring networks. The only pertinent reference in MBH98 was as follows:

Certain densely sampled regional dendroclimatic data sets have been represented in the network by a smaller number of leading principal components (typically 3–11 depending on the spatial extent and size of the data set). This form of representation ensures a reasonably homogeneous spatial sampling in the multiproxy network (112 indicators back to 1820). [our bolds]

This statement contains no reference to the use of Preisendorfer’s Rule N.

In connection with the calculation of temperature principal component series, a different calculation, MBH98 does refer to the use of Preisendorfer’s Rule N as follows:

a conventional Principal Component Analysis (PCA) is performed… An objective criterion was used to determine the particular set of eigenvectors which should be used in the calibration as follows. Preisendorfer’s selection rule ‘rule N’ was applied to the multiproxy network to determine the approximate number Neofs of significant independent climate patterns that are resolved by the network, taking into account the spatial correlation within the multiproxy data set.

Before trying to interpret these two statements from a text analytic point of view, I will make four quick points about rules for deciding the number of PCs to retain:

  • The briefest survey of PC literature will show that there are many approaches to selecting the number of PC series to retain and Preisendorfer’s Rule N is far from being a “standard selection rule”.
  • in fact, Urban, in a presentation about PCs cited on Jan. 6, 2005 by Mann at realclimate stated that the choice was subjective as follows: “It should be noted that because the goal of PCA is essentially utilitarian, the choice of how many axes to retain is ultimately subjective. In practice, either 2 or 3 axes are retained, simply because it is difficult to project more than this onto a printed page”.
  • Overland and Preisendorfer [1982] themselves argued that being significant under Rule N was only necessary for significance; they did not argue that it was sufficient.  
  • The real test for retaining a PC series is not whether it is significant under Preisendorfer’s Rule N (or some other such rule), but whether it is scientifically significant,. For example, Franklin et al. [1995] stated: “In the final analysis, the retained components must make good scientific sense (Frane & Hill 1976; Legendre & Legendre 1983; Pielou 1984; Zwick & Velicer 1986; Ludwig & Reynolds 1988; Palmer 1993).”

Now, from a text analytic perspective, a reasonable reader might conclude that the difference in description of the PC retention policy in the two cases – tree rings and temperatures – pointed to the use of different procedures in the two calculations. In fact, the form of PC calculation in the two calculations differed: we have determined that the temperature PC calculations were centered calculations, while, as we’ve pointed out in our recent articles (and earlier), the tree ring PC calculations were not conventional centered calculations. Mann et al. have recently (Jan. 6, 2005) acknowledged that they did not use a “standard centered method” so their use of an uncentered method is no longer in dispute.

Applying Preisendorfer’s Rule N to Tree Ring Networks

The real test for whether Preisendorfer’s Rule N was used in MBH98 was whether the actual number of selected PCs can be replicated using this method.

The actual retentions for each calculation step/network combination were not provided in MBH98, its SI or at Mann’s FTP site. The first complete listing of actual retentions came in the Corrigendum SI (July 2004). Even the Corrigendum SI contains no summarized listing: the following table was collated from the Corrigendum SI and shows the number of retained PCs by calculation step-network combination. (It was impossible to deduce this table with the additional disinformation of Mann et al. [2003] that 159 distinct series were used, since only 139 distinct series were actually used. Any such deduction attempts were further blocked by erroneous listings of the number of series used in the AD1450 step and the erroneous non-use of 6 available series in the AD1500 step. These do not affect early 15th century results, but frustrate attempts at replication.)

1400 1450 1500 1600 1700 1730 1750 1760 1780 1800 1820
Stahle/OK 0 0 0 0 3 3 3 3 3 3 3
Stahle/SWM 1 1 2 4 7 7 9 9 9 9 9
NOAMER 2 2 6 7 7 7 9 9 9 9 9
SOAMER 0 0 0 2 2 2 3 3 3 3 3
AUSTRAL 0 0 0 3 3 3 4 4 4 4 4
Vaganov 0 1 1 2 2 2 3 3 3 3 3
PC series 3 4 9 18 24 24 31 31 31 31 31
Direct proxy 19 21 19* 39 50 55 58 62 66 71 81
Total series 22 25* 28* 57 74 79 89 93 97 102 112

Table 1. Proxy series used in MBH98 (collated from Corrigendum SI, July 2004), showing the number of retained PC series by network-calculation step combination. *: The total number of series used in the AD1450 step is incorrectly stated in MBH98 as 24 (but error is not reported yet). Six series available in the AD1500 network are not used.

The first hints that a Preisendorfer-type policy had supposedly been used in MBH98 came in our Nature correspondence. In response to our observation of the error in their PC methods, Mann et al. [Revised Nature Reply] had noticed that, under correct PC calculations, the bristlecone pine pattern was demoted from the PC1 to the PC4.

precisely the same ‘hockey stick’ PC pattern appears using their convention, albeit lower down in the eigenvalue spectrum (PC#4) (Figure 1a). If the correct 5 PC indicators are used, rather than incorrectly truncating at 2 PCs (as MM04 have done), a reconstruction similar to MBH98 is obtained.

They argued that they could still salvage a hockey-stick shaped series using a Preisendorfer-type calculation on the AD1400 North American network. The calculation published on Nov. 22, 2004 at realclimate showing the implementation of a Preisendorfer-type calculation on the AD1400 North American network was originally submitted in our Nature correspondence. We had seen this diagram and calculation in August 2004 and had fully considered it in our GRL submission – in fact, it contributed to the approach taken in our GRL submission, which differs substantially from our previous Nature submission.

Realclimate Nov 2004 Figure 1

Figure 1 below, http://www.realclimate.org/index.php?p=9 (Nov. 22, 2004), and the two tables are all taken from realclimate, illustrating the application of the supposed Preisendorfer-type calculation. (The original section from Preisendorfer is re-typed here for reference.) The blue and red lines show the simulation results (using AR1 models of the AD1400 North American network) under MBH98 and centered PC calculations respectively; the red and blue points show actual results from the MBH98 and centered PC methods respectively. Preisendorfer’s Rule N selects PC series as long as the actual eigenvalue exceeds the simulation. For the MBH98 method, 3 eigenvalues are clearly separated under Rule N and perhaps 7 in a centered calculation. This result is strangely described by Mann et al as follows:

In the former case, 2 (or perhaps 3) eigenvalues are distinct from the noise eigenvalue continuum. In the latter case, 5 (or perhaps 6) eigenvalues are distinct from the noise eigenvalue continuum.

It seems obvious that the selection of 2 (rather than 3) eigenvalues in MBH98 cannot be directly justified on this diagram without appeal to some still unstated method.

Eigen # % Variance Cum % Variance
1 0.3818 0.3818
2 0.0976 0.4795
3 0.0491 0.5286
4 0.0354 0.564
Table 1 (from realclimate – MBH98 method.  Bold -retained PCs
Eigen # % Variance Cum % Variance
1 0.1946 0.1946
2 0.0905 0.2851
3 0.0783 0.3634
4 0.0663 0.4297
5 0.0549 0.4846
6 0.0373 0.5219
Table 2, from realclimate – centered “MM” method. bold – retained PCs
FIGURE 1. “Comparison of eigenvalue spectrum resulting from a Principal Components Analysis (PCA) of the 70 North American ITRDB data used by Mann et al (1998) back to AD 1400 based on Mann et al (1998) centering/normalization convention (blue circles) and MM centering/normalization convention (red crosses). Shown also is the null distribution based on Monte Carlo simulations with 70 independent red noise series of the same length and same lag-one autocorrelation structure as the actual ITRDB data using the respective centering and normalization conventions (blue curve for MBH98 convention, red curve for MM convention). In the former case, 2 (or perhaps 3) eigenvalues are distinct from the noise eigenvalue continuum. In the latter case, 5 (or perhaps 6) eigenvalues are distinct from the noise eigenvalue continuum.” Original legend from: Mann, http://www.realclimate.org/index.php?p=9 

Replication of Realclimate Nov 22, 2004 Figure 1

Figure 2 below shows my replication of the above calculations. The left panel repeats realclimate Nov 22, 2004 Figure 1 (as above), while the right panel shows my emulation, using the script here. The salient features of the methods are obviously captured.

FIGURE 2. AD1400 North American network – Preisendorfer-type calculations Left panel: Mann et al. [realclimate]. Points – NOAMER netowrk; lines – simulations. Blue – MBH98 decentered; red – centered. Right panel:  Emulation of calculation in left panel.

TESTING OTHER NETWORK/TIMESTEP COMBINATIONS

MBH98 has 6 networks with erratic changes of PC retention by timestep, yielding a total of 17 network/timestep combinations, all of which are examined below,

Stahle/OK

MBH98 only used one network/directory combination here, retaining three PCs. The observed retention is inconsistent with Rule N. The PC3 is insignificant under Rule N, but is retained anyway. There is little difference between MBH98 (blue) and centered (red) results – presumably because Stahle pre-whitens site chronologies.

Blue – MBH98 decentered; red – centered
Eigen # % Var Cum % Var
1 0.3908 0.3908
2 0.1532 0.544
3 0.076 0.62
4 0.0634 0.6834
5 0.0557 0.7391
6 0.0463 0.7854
MBH98 Eigenvalue Summary. Blue bold shows retained PCs. The retention of the PC3 is not consistent with supposed policy.

Stahle/SWM

In this network, there are 6 timesteps with different retained PCs. None of the retention patterns can be obtained through a direct application of Rule N.

  • the PC2 in the MBH AD1400 network qualifies under Rule N, but is not used
  • the PC2 in the MBH AD1450 network qualifies under Rule N but is not used
  • the PC3 in the MBH AD1500 network qualifies under Rule N but is not used
  • the PC4 in the MBH AD1600 network does not qualify under Rule N, but is used anyway;
  • the PC4-PC7 in the MBH AD1700 network do not qualify under Rule N, but are used anyway;
  • the PC4-PC9 in the MBH AD1750 network do not qualify under Rule N, but are used anyway;
  • the network does not change between AD1700 and AD1750 and thus Rule N would not yield a retention change. Nonetheless, two additional PCs are retained in the AD1750 step.
Blue – MBH98 decentered; red – centered
Eigen # % Variance Cum % Variance
1 0.4539 0.4539
2 0.3215 0.7754
3 0.1187 0.8941
4 0.0744 0.9685
5 0.0235 0.992
6 0.0463 0.7854
MBH98 Eigenvalue Summary. Blue bold shows retained PCs. The non-use of the PC2 is inconsistent with new policy. (The existence of this network is inconsistent as well, since there are only 3 sites (6 series) and 2 of the 3 sites appear to be either different versions of the same site or to contain a splicing error.
Blue – MBH98 decentered; red – centered
Eigen # % Variance Cum % Variance
1 0.5072 0.5072
2 0.2671 0.7743
3 0.1096 0.8838
4 0.0783 0.9621
5 0.0283 0.9904
6 0.0096 1
MBH98 Eigenvalue Summary. Blue bold shows retained PCs. The non-use of the PC2 is inconsistent with new policy. (The existence of this network is inconsistent as well, see AD1400).
Blue – MBH98 decentered; red – centered.
Eigen # % Variance Cum % Variance
1 0.4134 0.4134
2 0.1853 0.5986
3 0.1347 0.7333
4 0.0627 0.796
5 0.0553 0.8513
6 0.0437 0.895
MBH98 Eigenvalue Summary. Blue bold shows retained PCs. The non-use of the PC3 is inconsistent with new policy.
Blue – MBH98 decentered; red – centered
Eigen # % Variance Cum % Variance
1 0.385 0.385
2 0.1694 0.5544
3 0.0972 0.6516
4 0.0691 0.7206
5 0.0525 0.7731
6 0.0441 0.8172
MBH98 Eigenvalue Summary. Blue bold shows retained PCs. The use of the PC4 is inconsistent with new policy.
Blue – MBH98 decentered; red – centered
Eigen # % Variance Cum % Variance
1 0.3683 0.3683
2 0.1718 0.54
3 0.0945 0.6346
4 0.0574 0.692
5 0.045 0.737
6 0.0365 0.7735
7 0.0297 0.8032
8 0.0277 0.8309
9 0.0247 0.8556
10 0.0241 0.8797
MBH98 Eigenvalue Summary. Blue bold shows retained PCs in AD1700 network and purple bold the additional retained PCs in AD1750 network. The retention of PC4 and greater is inconsistent with the new policy as is the increase in the number of PCs in the AD1750 network.

South America

In the South American network, the PC3 is retained for calculation steps after AD1750, but does not qualify under Rule N.   (This network was also the source of major inconsistencies between the listed sites and the sites actually used (see Corrigendum), as the network was reduced from 18 sites to 11 sites. The reasons for the inconsistency provided in the Corrigendum are incorrect, as I’ll post on another occasion.)

Blue – MBH98 decentered; red – centered
Eigen # % Variance Cum % Variance
1 0.379 0.379
2 0.1528 0.5318
3 0.1145 0.6463
4 0.0832 0.7294
5 0.0681 0.7975
6 0.0509 0.8484
MBH98 Eigenvalue Summary. Blue bold shows retained PCs in AD1600-1730 calculation steps; purple bold in AD1750+ calculation step. The retention of the PC3 is inconsistent with the new policy. The addition of a PC to a network in a later step is also inconsistent with the policy.

Australia/NZ

The retention of the PC3 in the AD1600-1730 steps and the retention of the PC3 and PC4 in the AD1750+ steps is inconsistent with Rule N.

Eigen # % Variance Cum % Variance
1 0.2646 0.2646
2 0.1682 0.4328
3 0.0974 0.5303
4 0.0854 0.6156
5 0.0764 0.6921
6 0.064 0.7561
MBH98 Eigenvalue Summary. Blue bold shows retained PCs in AD1600-1730 calculation steps. The retention of the PC3 is inconsistent with the new policy.
Eigen # % Variance Cum % Variance
1 0.3074 0.3074
2 0.1724 0.4798
3 0.0871 0.5669
4 0.0702 0.6371
5 0.069 0.7061
6 0.0611 0.7672
7 0.045 0.8122
MBH98 Eigenvalue Summary. Blue bold shows retained PCs in AD1750+ calculation steps. The retention of the PC3 and PC4 is inconsistent with the new policy.

Vaganov

The Vaganov network is a network of Russian sites. There is relatively little difference between centered and uncentered methods in this network. This appears to be because of pre-whitening used by Vaganov in developing tree ring chronologies. The pre-whitening dramatically reduces the autocorrelation in the network. These graphs are very important in assessing MBH98 practices, as they show series with large Preisendorfer significance, which are not used: the PC2 in the AD1450 network, the PC3-5 in all steps from AD1600 on.

Eigen # % Variance Cum % Variance
1 0.3548 0.3548
2 0.1914 0.5461
3 0.1033 0.6494
4 0.0756 0.725
5 0.0659 0.7909
MBH98 Eigenvalue Summary. Blue bold shows retained PCs in AD1450-1500 calculation steps. The non-use of the PC2 is inconsistent with the new policy.
Eigen # % Variance Cum % Variance
1 0.2093 0.2093
2 0.1782 0.3876
3 0.1205 0.5081
4 0.0544 0.5625
5 0.0433 0.6058
MBH98 Eigenvalue Summary. Blue bold shows retained PCs in AD1600-1730 calculation steps. The non-use of the PC3-PC5 is inconsistent with the new policy.
Eigen # % Variance Cum % Variance
1 0.2541 0.2541
2 0.1437 0.3977
3 0.1226 0.5203
4 0.0562 0.5765
5 0.0372 0.6137
6 0.0331 0.6468
MBH98 Eigenvalue Summary. Blue bold shows retained PCs in AD1450-1500 calculation steps. The non-use of the PC3-5 is inconsistent with the new policy.

North America after 1400

Again, there are inconsistencies in the implementation of the supposed policy: a) the non-use of the PC3 and perhaps PC4 in the AD1450 step; b) the use of the PC5-PC6 in the AD1500 step; c) the increase in retained PCs (within the same network) from the AD1450 step to the AD1500 step; d) the non-use of the PC8 in the AD1600 step.

Eigen # % Variance Cum % Variance
1 0.3567 0.3567
2 0.1032 0.46
3 0.0432 0.5032
4 0.0319 0.5351
5 0.0276 0.5626
6 0.0261 0.5888
7 0.0203 0.609
8 0.0195 0.6286
MBH98 Eigenvalue Summary. This network is used in the AD1450 and AD1500 calculation steps. Blue bold shows retained PCs in the AD1450 calculation step; purple bold the additional PCs retained in the AD1500 calculation step. The non-use of the PC3-4 is inconsistent with the new policy, as is the use of the PC5-6 in the AD1500 step. The change in usage between steps is also inconsistent.
Eigen # % Variance Cum % Variance
1 0.203 0.203
2 0.1264 0.3294
3 0.0473 0.3768
4 0.0295 0.4063
5 0.0257 0.432
6 0.024 0.456
7 0.0223 0.4783
8 0.0214 0.4996
9 0.0191 0.5187
MBH98 Eigenvalue Summary. Blue bold shows retained PCs in AD1600 calculation steps. The non-use of the PC8 appears inconsistent with the new policy.
Eigen # % Variance Cum % Variance
1 0.1755 0.1755
2 0.1298 0.3053
3 0.0509 0.3562
4 0.0377 0.3939
5 0.0304 0.4243
6 0.0266 0.4509
7 0.0251 0.476
8 0.023 0.499
9 0.0215 0.5205
10 0.0201 0.5406
11 0.0192 0.5598
MBH98 Eigenvalue Summary. Blue bold shows retained PCs in AD1750+ calculation steps.

Summary

The following table summarizes the inconsistencies between the observed PC retentions and the retentions according to Rule N.  It is obvious that application of the Preisendorfer Rule N method to actual networks does not yield the PC selections archived at the Corrigendum SI. In some cases, more PCs are archived; in other cases, fewer PCS: there is no obvious pattern. In addition, in 3 cases, different selections were made from the same network, e.g. 7 PCs were selected from the AD1700 SWM network in the AD1700 calculation; the same network was used in the AD1750 calculation step, but this time 9 PCs were selected. This would not be permitted without some still unreported adaptation of the method

Retained PCs
Network/Step Reported Emulated
OK/AD1700 3 2
SWM/AD1400 1 2
SWM/AD1450 1 2
SWM/AD1500 2 3
SWM/AD1600 4 3
SWM/AD1700 7/9 3
SOAMER/AD1600 2/3 2
AUSTRAL/AD1600 3 2
AUSTRAL/AD1750 4 2
VAGANOV/AD1450 1 2
VAGANOV/1D1600 2 5
VAGANOV/AD1750 3 5
NOAMER/AD1400 2 3
NOAMER/AD1450 2/6 4
NOAMER/AD1600 7 8
NOAMER/AD1750 9 10

.

Perhaps there is some common factor to the above process that we have not discerned – however, we are confident that no other third party in the world has been able to discern the pattern. Had Mann et al. archived their source code for these calculations (and for other calculations), then these issues would not be a matter of speculation.

Notwithstanding all of the above, as far as I’m concerned, the main issue is whether the PC series so selected are significant in a scientific sense, rather than a data mining sense. We’ve provided many caveats in our E&E article to reliance on the bristlecone pine series as arbiters of world temperature history – whether they are in a PC1 or a PC4. However, if Mann is to insist at this late stage that the selection of 5 PCs is justified on the present record, it seems evident to me that an explanation is required for exactly how the Preisendorfer-policy set forth here can be reconciled with actual retentions. Perhaps it can, but I’ve so far been unable to figure out the secret. These guessing games are also pointless. I invite any readers that have got this far should express their objections to the U.S. National Science Foundation and to Nature that this important source code should continue to remain undisclosed.

References:

Franklin, Scott B., Gibson, David J., Robertson, Philip A.,Pohlmann, John T. and Fralish, James S. (1995), Parallel Analysis: a method for determining significant principal components, Journal of Vegetation Science 6: 99-106.

Overland and Preisendorfer [1982], “A Significance Test for Principal Components Applied to a Cyclone Climatology”, Mon. Wea. Rev. 110, 1-4.

Footnote: Preisendorfer’s Rule N is a simulation method based on white noise, stated as follows:

This rule of PC selection is a dominant-variance rule and is based on a Monte Carlo procedure which simulates sampling from Np(0,Σ), with  Σ=σ2 Ip. The null hypothesis is that our n x p data matrix Z has been drawn from such a population. By following the procedure outlined below, we can systematically accept or reject this hypothesis. Let R be the centered random data set so formed (cf 5.4 and 5.5). Forming S=RTR for each such sample, we then build up a cumulative distribution for each of the ρ = min(n-1,p)) non-zero eigenvalues λj, j = 1,…,ρ. We can then compare the data eigenvalues of the given n x p data set Z, one by one, with these cumulative distributions. The details follow.

Construct (say) 100 independent realization of each of np variates from N(0,1). Form the n x p matrix R as in (5.4) and (5.5). This is the random n x p counterpart R to the given n x p data matrix Z. The ωth realization R(ω) of the centered R results in an ordered sequence of non-zero eigenvalues:

λ1(ω) > …> λρ(ω), ω = 1, …,100, ρ = min(n-1,p).

Write

Uj(ω) = λj(ω) {  ρ-1 Σk=1:ρ λk(ω)   }–1,  ω=1:100, j=1:ρ

For each j, order these (after relabelling) as

Uj(ω1) < …U j (ω100)

and set

σj(05) = Uj(ω5) ;  and σj(95) = Uj(ω95) ;

These σj values define the 5% and 95% points on the cumulative distribution for the jth random eigenvalues.

 For the given data matrix Z with its associated ordered set of non-zero eigenvalues, write:

Vj = dj {  ρ-1 Σk=1:ρ dk }-1 j= 1,…ρ

Thus we have

          Rule N: p’ is the greatest j for which Vj > σj(95); 0 if no such j exists.

he random n x p counterpart R to the given n x p data matrix Z. The ωth realization R(ω) of the centered R results in an ordered sequence of non-zero eigenvalues:
… > \lambda_p(\omega)” alt=”\lambda_1(\omega) > … > \lambda_p(\omega)” align=”absmiddle” width=”165″ height=”22″> , ω = 1, …,100, ρ = min(n-1,p).

Write
, ω=1:100, j=1:ρ
For each j, order these (after relabelling) as

and set
; and  ;
These σj values define the 5% and 95% points on the cumulative distribution for the jth random eigenvalues.For the given data matrix Z with its associated ordered set of non-zero eigenvalues, write:
j= 1,…ρ
Thus we have
Rule N: p’ is the greatest j for which Vj > σj(95); 0 if no such j exists.

By Stephen McIntyre | Posted in MBH98 | Tagged | Comments (3)

Bre-X #3: Core and Code

Feb 6, 2005 – 10:42 PM

In the aftermath of the Bre-X fraud, quite naturally, there was a great deal of examination of procedures, methods and due diligence.

One of the standard – actually it would be more accurate to say universal – practices in mineral exploration is that diamond drill core is split in half on site. The core is cylindrical and the split is in two half-cylinders. At or near the mine, the retained half is placed in drill boxes, which are like trays, and the trays are placed in special core racks. At various points in the financing of the exploration and development of a mining project, consulting geologists or engineers will visit the project and inspect the core. For ore body evaluation, it is important not just to know the assay grades, but to see the alteration of the rocks in the zone, etc. Even at a remote location like Bre-X, in the back of Indonesia, there would have been visitors to the exploration project.

But visitors to Bre-X were not able to see any split core, because Bre-X said that the metallurgy of the ore was such that they needed the entire core to get an accurate sample. I don’t think that the investing public at large had any idea that there was no drill core at the Bre-X site. I was in the exploration financing business in Toronto, a city where exploration financing is a big activity, and I’d never heard about this before the collapse. I cannot comprehend why the analysts for any of the brokerage houses which did financings for Bre-X would not have run for the exit when there was no drill core at the project site.

But there are many such instances, where people get caught up in the excitement. The analysts probably felt that, if they didn’t do the financing, some one else would; and there were big commissions in the financings. There can be a madness of crowds, which people in stock markets are familiar with.

Now source code in multiproxy climate studies is not the same as drill core and the major concern is mistakes, rather than fraud. But it is incomprehensible to me why the climate science community does not routinely require the archiving of data as used and source code as used, as an elementary means of quality control – just as exploration projects keep drill core.

At one time, when dendrochronologists were writing for the Tree Ring Bulletin with a circulation of (say) 100, maybe it wasn’t necessary; but now there’s real money involved in climate policy. Something equivalent to drill core should be routinely available.

At the present moment, Mann is arguing that the errors identified to date in MBH98 do "not matter" and that he can "get" the similar results using different methods. Obviously I disagree with this. But here I’m pondering the continued unavailability of MBH98 code; despite all the publicity, Mann obdurately refuses to produce his source code. If I were in his shoes, I would have released the code long ago, just to be rid of the issue. I am baffled as to why climate scientists don’t rise up and demand that he produce his source code before considering a single additional claim or argument by Mann et al.

By Stephen McIntyre | Posted in Disclosure and Diligence | Tagged | Comments (5)

Spot the Hockey Stick! #1 in a series of many

Feb 6, 2005 – 8:27 PM

Steve and I thought it would be a great idea to play a game of “Where’s Waldo?” Except that instead of finding a cartoon character on a page in a children’s book featuring lots of cartoon characters, we’d instead try to spot the “Mann Hockey Stick” hiding in news articles and scientific reports.

Our first examples are from the BBC, staunch promoter of the idea of climate change being caused by humans, and publisher of the most extreme results of climate models as if they were actual scientific data.

Curiously some of the articles that I distinctly remember were present on the site appear to be missing. It’s as if the Hockey Stick is causing a black hole in the BBC’s collective memory (or mine). But still, there are plenty of examples, for Hockey Stick watchers to enjoy:

Example 1


Example 2: Sir John Houghton of the IPCC pictured in front of…guess what?


Example 3: click on “Long Term High”

bbchockeystick.JPG

If there are correspondents out there with even better examples, please fill out the comment field below.

By John A | Posted in Spot the Hockey Stick! | Comments (1)

On the value of skepticism in science

Feb 6, 2005 – 8:04 PM

I throw this in for reasons that should be obvious.

“The fallibility of methods is a valuable reminder of the importance of skepticism in science. Scientific knowledge and scientific methods, whether old or new, must be continually scrutinized for possible errors. Such skepticism can conflict with other important features of science, such as the need for creativity and for conviction in arguing a given position. But organized and searching skepticism as well as an openness to new ideas are essential to guard against the intrusion of dogma or collective bias into scientific results.”

– “On Being a Scientist: Responsible Conduct in Research”, National Academy of Sciences, 1995

By John A | Posted in General | Comments Off on On the value of skepticism in science

Bristlecone "Adjustment" #1

Feb 6, 2005 – 6:04 PM

There has been some recent chatline discussion on whether the MBH98 bristlecone pine series were “adjusted” for non-climatic factors. A number of posters seem to think that they were so “adjusted. For example, Tom Rees said on http://www.davidappell.com:

on Monday, January 31st, Tom Rees said

Me again… James: Mann et al 1999 seem to be very clear. They’re not sure what causes the surge in the Bristlecone pines, so they play safe and remove it. So it’s not in the ‘Hockey stick’ at all…

To the extent that one can ever be certain about MBH98 methods, I am completely confident that the “adjustment” discussed in MBH99 was not implemented in MBH98. The archived MBH98 PC series can be replicated almost exactly and do not show any such “adjustment”; an email to Natuurwetenschap by Mann states that the sensitivity on bristlecones was done after MBH98.

This does not totally end the story. Curiously, the archived version of the North American PC1 at NOAA for MBH99 is not the “adjusted” version, but the unadjusted version from the BACKTO_1000 directory. In most walks of life, there would be some check that they did not forget to do the adjustment in their downstream calculations, but I am unaware of any such checks, and, the unavailability of source code or supporting calculations makes direct verification impossible.

Secondly, one of the representations of MBH98 was that the proxies were all linearly related to climatic factors. The discussion of MBH99 states quite clearly that non-climatic factors affect bristlecone pine growth and adjustments are required. (I disagree with the MBH99 adjustment, but will return to that with a full description of the procedure on another occasion in the near future.) If so, then this clearly imposes an obligation on MBH to have promptly notified Nature of the fact that some of the MBH98 proxies did not meet the criteria set out and to publish a Corrigendum, demonstrating the effect on MBH98 calculations.

Shouldn’t MBH “adjust” the bristlecone pines before they do their PC calculations for the 15th century network? Shouldn’t they have done this a long time ago? The bristlecones have already been demoted to the PC4 using correct PC calculations? If they are “adjusted” according to MBH99 methods, do they fall still further? Is there any change to their “significance”? Inquiring minds want to know.

I didn’t mention this issue in my first remarks on the current efforts to salvage MBH98, but it seems it might be an important issue and that MBH have little more homework to do.

By Stephen McIntyre | Posted in MBH98 | Tagged , | Comments (8)

Thompson # 1: Inconsistent Dunde Versions

Feb 6, 2005 – 4:19 PM

The Dunde (Himalayas) ice core is a staple of multiproxy studies (MBH98, Crowley and Lowery [2000], Yang et al [2002] used in Jones and Mann [2003], Jones and Mann [2004]). It was one of 4 series that contributed the only hockey-stick-ness to the results – see here . Although the core was taken over 17 years ago, it has not been archived at WDCP. In 2004, after I objected to Climatic Change, Thompson archived a decadally averaged version, which is inconsistent with the "grey" version used in MBH98 (see below) and both are inconsistent with the 50-year smoothed version used in Yang et al. [2002]. Thompson has been funded by the U.S. National Science Foundation. This matter should be resolved by prompt archiving of original data, together with a reconciliation of the different versions. (This applies to Guliya and Dasuopu as well.) Dunde: Thompson and MBH98 Versions MORE

By Stephen McIntyre | Posted in MBH98, Multiproxy Studies, Thompson | Comments (1)

Jacoby #1: A "Few Good" Series

Feb 6, 2005 – 7:48 AM

Jacoby is on the Hockey Team. His treeline temperature reconstruction was made by picking the 10 most "temperature-influenced" of 36 sites studied. Only these 10 sites were archived. I sought information on the other 26 through Climatic Change, the publishing journal. Jacoby refused, stating:

The inquiry is not asking for the data used in the paper (which is available), they are asking for the data that we did not use.

Imagine this argument in the hands of a drug trial. Let’s suppose that they studied 36 patients and picked the patients with the 10 "best" responses, and then refused to produce data on the other 26 patients on the grounds that they didn’t discuss these other patients in their study. It’s too ridiculous for words. Yet Climatic Change saw no problem with this refusal. Jacoby went on to say that his research was "mission-oriented" and that:

As an ex- marine I refer to the concept of a few good men. A lesser amount of good data is better without a copious amount of poor data stirred in.

Imagine ex-marines with this philosophy in charge of drug trials. Maybe they already are.


Jacoby Response to Data Request
Jacoby and d’Arrigo [Clim. Chg. 1989], together with d’Arrigo and Jacoby [1992], is a temperature reconstruction, which is applied in many multiproxy studies (e.g. Jones et al [1998], 11 series used individually in MBH98, as an "adjustment" to the North American PC1 in MBH99, Jones and Mann [2004]). Jacoby is a member of the Hockey Team.

Jacoby and d’Arrigo [1989] states on page 44 that they sampled 36 northern boreal forest sites within the preceding decade, of which the ten "judged to provide the best record of temperature-influenced tree growth" were selected. No criteria for this judgement are described, and one presumes that they probably picked the 10 most hockey-stick shaped series.

I have done simulations, which indicate that merely selecting the 10 most hockey stick shaped series from 36 red noise series and then averaging them will result in a hockey stick shaped composite, which is more so than the individual series. The process is not dissimilar to what happens in the MBH98 PC1. In the MBH98 PC1, the 14 most hockey stick shaped series account for over 93% of the variance. There is very little difference in appearance between a simple average of these 14 series and the EOF-weighted composite (PC1).

I was interested in testing whether Jacoby’s selection process imparted a bias to the data set under consideration. In order to test whether Jacoby’s selection of the 10 most "temperature-influenced" series had any significance relative to comparable selection from red noise series, I looked for the archived information of the 36 sites. I had previously located the 10 most "temperature-influenced" sites at WDCP archives (and have elsewhere discussed inconsistencies between this archive and the versions used in MBH98), but was unable to locate archived versions of the other 26 series.

As a result of some previous exchanges with Climatic Change (which I will probably discuss on another occasion), they adopted a policy in which authors were required to provide supporting data, but decided not to adopt a policy requiring authors to provide source code. Under this policy, I asked them to obtain the other 26 series from Jacoby (since I had had no success directly).

Jacoby refused to provide the 26 series and I found his reasoning as set out to Climatic Change quite interesting (my bolds).

The inquiry is not asking for the data used in the paper (which is available), they are asking for the data that we did not use. We have received several requests of this sort and I guess it is time to provide a full explanation of our operating system to try to bring the question to closure.

Speaking for myself and immediate colleagues who have been involved with my research: Most of our research has been mission-oriented, dendroclimatic research. That means to find climatically-sensitive, old-aged trees and sample them in order to extend the quantitative record of climatic variations. Also, to relate these records to the real world and investigate the climate system and its functioning.

The first part produces absolutely-dated time series of tree-ring variations. We try to sample trees at sites where there is likely to be a strong climatic signal, usually temperature or precipitation. Sometimes we are successful, sometimes we are not. We compare the tree-ring series to climate records to test what the climate signal is. We sample latitudinal treeline and elevational treeline looking for temperature-sensitive trees with both a high-frequency and low-frequency response to temperature. A high-frequency temperature response to summer is most frequently found at these extreme locations. However, trees have much more information if one finds trees with a good communal high and low frequency variations that correspond or correlate to local or regional temperatures for longer seasons. There is abundant information to explain the physiological processes in cooler seasons and why trees can respond to more than just summer season. The sampling and development of a tree-ring chronology is an investment of research energy, time, and money.

The best efforts in site selection and sampling do not always produce a good chronology. It is only as the samples are processed and analyzed that the quality, or lack thereof becomes evident. First is the dating: this is enabled by high-frequency common variation among the trees. The dating is achieved and tested by various methods. Then the chronology is developed from the correctly dated ring-width measurements and evaluated. Testing: Is there a common low-frequency signal among the trees? At a good temperature- sensitive site with good trees, there is. We conduct common period analyses of the low- frequency variation within the cores samples from a site.

Sometimes, even with our best efforts in the field, there may not be a common low-frequency variation among the cores or trees at a site. This result would mean that the trees are influenced by other factors that interfere with the climate response. There can be fire, insect infestation, wind, or ice storm etc. that disturb the trees. Or there can be ecological factors that influence growth. We try to avoid the problems but sometimes cannot and it is in data processing that the non-climatic disturbances are revealed.

We strive to develop and use the best data possible. The criteria are good common low and high-frequency variation, absence of evidence of disturbance (either observed at the site or in the data), and correspondence or correlation with local or regional temperature. If a chronology does not satisfy these criteria, we do not use it. The quality can be evaluated at various steps in the development process. As we are mission oriented, we do not waste time on further analyses if it is apparent that the resulting chronology would be of inferior quality.

If we get a good climatic story from a chronology, we write a paper using it. That is our funded mission. It does not make sense to expend efforts on marginal or poor data and it is a waste of funding agency and taxpayer dollars. The rejected data are set aside and not archived.

As we progress through the years from one computer medium to another, the unused data may be neglected. Some [researchers] feel that if you gather enough data and n approaches infinity, all noise will cancel out and a true signal will come through. That is not true. I maintain that one should not add data without signal. It only increases error bars and obscures signal.

As an ex- marine I refer to the concept of a few good men.

A lesser amount of good data is better without a copious amount of poor data stirred in. Those who feel that somewhere we have the dead sea scrolls or an apocrypha of good dendroclimatic data that they can discover are doomed to disappointment. There is none. Fifteen years is not a delay. It is a time for poorer quality data to be neglected and not archived. Fortunately our improved skills and experience have brought us to a better recent record than the 10 out of 36. I firmly believe we serve funding agencies and taxpayers better by concentrating on analyses and archiving of good data rather than preservation of poor data.

I guess I won’t be getting the data. It would be my position that, if they picked 10 of 36 sites, they used all 36 sites in their study. Imagine this argument in the hands of a drug trial. Let’s suppose that they studied 36 patients and picked the patients with the 10 best responses, and then refused to produce data on the other 26 patients on the grounds that they didn’t discuss these other patients in their study. It’s too ridiculous.

By Stephen McIntyre | Posted in Archiving, Jacoby, Multiproxy Studies, Proxies | Tagged , | Comments (12)