I discussed Fisher’s Greenland dO!8 proxy yesterday and thought that it would be interesting to discuss its particular function in hockey stick manufacture in Jones et al 1998. Each hockey stick is, after all, made by a master craftsman. The Greenland dO18 is one of only 10 series in Jones et al 1998. Let’s see if we can figure out exactly what its particular function in making a good hockey stick and why it was chosen.
One important technique in making hockey sticks is the splice. This can occur in many different forms. Most people have been distracted by petty issues like splicing instrumental and proxy records and failed to appreciate the artistic splicing that occurs between short and long series, which are integral to one style of hockey stick manufacture as practiced by one group of artisans.
To illustrate this, I’ve selected the Greenland dO18 series (available in the MWP) and 2 shorter series from Jones et al 1998. This is in about the right proportion to the 10 series used in the original, where there are only 3 series are available at the start of the 11th century and only 4 prior to 1400. The other 2 series (Kameda melt, also from Greenland and the Jacoby NH reconstruction) obviously have a very different appearance from the Greenland dO18. It takes a discerning eye to "understand" that all 3 series are a temperature "signal" plus white noise, despite their differing appearance. (If you are not a member of the Hockey Team, you will not understand this. I am not one of the elect who can recognize the common signal in all 3 series.)
But you can see one big difference: the long MWP has no centennial variation and just looks like noise on a centennial scale, while the two short series have a lot of centennial variation and much more autocorrelation.
Following Jones, all series are standardized on the 1901-1950 intervals. Trenberth 1984 pointed out within climate literature the well-known statistical point that means and standard deviations calculated on short segments of autocorrelated series may not be very accurate and the longest possible record should be used. However, this is contrary to usual Hockey Team short-segment standardization and was not done here either. You can see that, when one series has little centennial fluctuation and the others do, their relative appearance will differ quite remarkably depending on which 50-year period was used for standardization.
Figure 1. Three of 10 Jones et al 1998 series. Red – smoothed with 21 -year gaussian filter.
I’ve been able to replicate the general appearance of Jones’ results using a full data set, but not exactly. In the figure below, I’ve applied the Jones’ method as best as I can to a data set consisting of only the above 3 series, hoping that this can isolate important aspects of form and function. I’ve done this in two steps. The middle panel shows the simple average of the 3 series, which already captures some of the main features of the Jones hockey stick, but with an extremely noisy MWP.
The bottom panel shows the effect of a "variance stabilization" method (copyright Hockey Team) on this 3-series network as presented by noted statisticians Briffa and Osborn in the noted statistical journal Dendrochronologia. This seminal article has been applied by other hockey players, but has been sadly neglected by applied statisticians and workers in other disciplines.
If you have a network of white noise, the variance of the mean is 1/sqrt(N). So in the early portion of this network with only 1 series (or 3 of 10 in the full network), the variance in the MWP blows up. Briffa and Osborn calculate the average interseries correlation- which is about 0.15 or so at best – and use this to calculate the "effective" number of series relative to the "effective" number of series when there is a full network. In this small subset, the "effective" number of series in the last portion is a shade under 3 (interseries correlation of 0.17) and is only 1 "effective" series in the first portion. So following this procedure, the variance of the portion of the network with only 1 series is reduced to reflect the "effective" number of series. I’ve figured out the main steps, but haven’t got this step exactly. I’ve asked Jones for code and exact data so that I could exactly replicate his results, but guess what – he refused.
What is rather pretty about showing this with only 3 series is how it lays bare rather nicely a couple of the key components of the Jones hockey stick and how it achieves a rather good emulation all by themselves. You can directly see how the Greenland dO18 series carries into the MWP portion of the shaft – which is noisy and featureless; while the Jacoby NH series with its extreme lack of high-frequency character imprints the second half.
Figure 2. Top – archived version. Middle – 3 series version pre-stabilization. Bottom – 3-series version, post-stabilization.
I’ve posted on the other two Jones et al 11th century proxies – Briffa’ s Polar Urals and Tornetrask series -at length here – look at the Jones et al 1998 Category on the right. There’s nothing presented here that doesn’t apply to them. If anything, they have their only little tweaks to "improve" the hockey stick – little "adjustments" showing the handwork of master artisans. A connoisseur should be able to identify a Jones-type series by its craftsmanship, which differs from Mannian craftsmanship: a long featureless MWP; centennial variation initiating in the LIA; modern values squeaking out a bit higher than the MWP.
In fact, we get the key elements from only two series – the Greenland series and the Jacoby series. as shown below:
Figure 2. Top – archived version. Middle – 2 series version pre-stabilization. Bottom – 2-series version, post-stabilization.
Of course, just showing the proxies by themselves leaves the hockey stick unfinished. Indeed, if one looks at the proxy portion of the Jones HS by itself, it’s possible that someone might say: so what. So the master craftsman then puts the instrumental record in a bold flourish and voila, ready to go out on the rink.
Climate Audit 
47 Comments
Obviously, I am slow on the uptake, but have you tried using the “verification” periods (e.g. 1850-1900) for standardisation, and the “calibration” periods (e.g. 1901-1950) for verification ? Does that make a big difference, and does sensitivity to that sort of analysis show lack of robustness ?
Yet more fascinating insights into free exchange of methods ! I do wonder what the NAS panel will be saying on that 🙂
per
The standardization doesn’t make sense to me. It seems like there is a logical fallacy there.
How can you reduce the variance like that if that’s not what the data shows. Sure with less data, things may look more variable (like a portfolio of stocks with only one stock versus some non-correlated stocks giving portfolio risk reduction).
If what I’m saying is right, why don’t you publish a little note explaining this? Emphasis on clarifying the stat method more then reconstruction specifics.
#1. I’m going to do this on some of the gridcell correlations in Osborn and Briffa next. For people that condemn the r2 statistic, they use correlation r to include/exclude proxies. I’m curious to see how things like Yamal turn up. These things always take more time than you think because Osborn and Briffa used a different temperature data set for their Science article than they said they did. But they don’t think that this makes enough of a difference to issue a Corrigendum. But I’m slow finishing these things because I stray off into byways like Greenland while I’m collecting my data. If Yamal passes an RE statistic, I’d be amazed.
If you look at the amplitude of the Jacoby NH series and then the amplitude of the reconstruction, you’ll see that the shapes are similar but the amplitude of the reconstruction is attenuated relative to the amplitude of the JAcoby NH series by blending of white noise type series. Remind you of anything?
#1. I’d be amazed if the NAS panel touches that issue or any other issue that would possibly be embarassing to the Team.
Just out of interest, the NAS committee started 19/01/06, and was scheduled to produce a report in four months. They had meetings in april and may; have you heard any indication of when something might transpire ?
per
A small point, but half-seriously offered: wouldn’t it be more correct to call these manipulations of the series “laminations” rather than “splices” as they are a stacking of chronologically overlapping records instead of a concatenation of series? The statistical justifications for doing each of them should be different so the terminology ought to be different.
#7. Nice point. If you "laminate" your proxy set, re-do your statistical model in steps, and then compose your final answer in steps a la MBH, would you call that a splice or a lamination? It’s a bit of both.
#6. No news or rumors that I’ve heard.
Posted for Kevin Gray:
I’m curious if anyone has come across an example of
Box-Jenkins Transfer Function modeling being done on
climate data. I have looked for something I could use
in some internal company training as a general example
but have come up empty.
Regards,
Kevin Gray
The Knowing One does not know what is meant here. If a series is a random walk (i.e. increments are white noise), then the variance of the mean is N. What am I missing?
Is there a general method that is proposed for combining series of different extent? Is stepwise reconstruction really needed? Can’t you combine them all and just have a higher uncertainty in the periods that have less data or more observed variance (probably because of less “portfolio-based”* variance reduction?
I forget the technical term, but this is a finance 101 MBA topic. Steve or the econ prof (Ross) can fill it in.
Steve:
I’ve been following CA for some time now, and am grateful for the contribution you’ve been making to the issues at hand. I realize that you run this blog purely from personal interest; a hobby, if you will. Having said that, I wonder if it isn’t useful to re-examine what you’re trying to accomplish and the best method of achieving it.
Although you’ve done a great job of exposing the myriad flaws in the hockey stick (be it the original or the many variants produced by the hockey team), it remains an icon of AGW theory both in the popular mind and in the agencies (ie IPCC) that produce policy recommendations.
CA has made, and continues to make, an important point. At the same time, like it or not, the peer-reviewed body of published literature remains the primary means by which science is injected into the policy process. If one of your aims in running CA is to reduce or eliminate the weight given to the HS, the only real way of doing so is to publish more of your criticisms in accepted journals.
By restricting your criticism to the pages of CA, you are deemed essentially irrelevant by much (most?) of the scientific community and, worse, the policy wonks. Your work, regardless of its merits, is largely ignored or written off as the ravings of a contrarian / denialist / oil company shill / whatever.
I suggest that CA would achieve greater credibility, and would function much more effectively in disseminating information and generating debate within the climate community, if you published several more papers focusing on the specific shortcomings of the HS. Since most of the criticism is statistical in nature, papers could be submitted not only in climate journals but in statistical ones as well. Examination and validation of your work by professional statisticians would make your criticisms vastly more difficult to ignore. Such validation would also lend more weight to your other campaigns (Proper archiving! Update the proxies! Due diligence!) which will be important considerations in future developments, such as detection and attribution studies.
To summarize: I like CA. I think you do excellent work and I agree with most of it. I further think that if you want to maximize your impact on the climate change debate, you should publish and allow CA to function as a companion to a larger body of published work. You’re a businessman so think of it this way. By virtue of their publishing record, Mann et al have an excellent “brand” that gives them instant credibility (deserved or not). Your brand is neither well-known nor as highly regarded as it should be. The best way to “market” your ideas to the science and policy communities is through journals.
Best regards;
#12
Steve,
I also think this is a good idea, although I would not be so confident that the peer-review process in the area of climate research is nowadays always fair. But perhaps in statistics journals the potential reviewers may be still unbiased.
eduardo
#10: I think it should have been 1/sqrt(n). If x~N(0,1) then x-bar~N(0,1/sqrt(n)) where n is sample size.
Steve: sorry about the sloppiness – that’s obviously what I intended from the comment. I’ll edit.
#12, 13. I agree in principle and have some things in progress. It’s not that I feel frustrated with journals – I don’t.
Partly I’ve been looking for a good topic. Lot’s of what I do are small topics, but some small topics would be perceived as an (undeserved) admission of failure on the larger topics.
Also, I’ve been very frustrated in the “other studies” by the same or worse litigation problems in getting data and methods. Right now, for example, I don’t even know what sites were used in Briffa et al 2001. Not for lack of trying. Until February and late March of this year, I had no data from Esper or even any idea how his data or methods were done. I had missing pieces from Moberg until February. D’Arrigo et al 2006 is a sinkhole of missing data. However, some of the information that I’ve got this year has really given me a foothold on these other studies that I didn’t have before. The Polar Urals update from Esper has been very important.
I’m pretty sure that I can now show the non-robustness of the entire corpus of work (except Moberg) to this substitution and the bristlecones. Moberg falls apart on other grounds. This might be an interesting topic for a more formal study. The trouble is that it’s hard to do in 4 GRL pags and I don’t want to spend 2 years in limbo at one of the journals with more space.
I could write up some of my notes on replication. I’ve got a standing invitation from a journal to do that. An article on MBH replication would be interesting and I’ve got it mostly written.
I’d also like to write up the algebra of MBH. I mentioned a couple of years in MM03 (contra Eduardo) that MBH could be represented as a linear weighting of proxies. I’ve implemented that as an output of my algorithm and the weights are highly interesting to look at. Maybe that would fly.
From the MBH algebra, I can show that it’s partial least squares, derive some properties about variance attenuation for linear estimators, wade through the noise properties of the MBH proxies and reconcile the VZ issues. This is something that’s topical right now. There are aspects of this that I wasn’t in a position to deal with last year, as I’ve had to improve my background in what PLS was, once I figured out that MBH could be fit into that mould. That would be interesting, because some of the issues with this method carry over into Mann and Jones 2003 and Hegerl et al 2006.
Eduardo sent me output from the Echo-G model. I’ve got some thoughts on that.
I’ve done some interesting analyses on tree ring chronologies and can derive some of their methods using linear mixed effects methods. I did that about 2 years ago and haven’t pursued it, but it’s interesting.
I wake up every day with 3 new ideas. The advantage of writing little notes to the blog is that it serves as a bit of a diary, so I don’t lose track of them. If I didn’t use the blog as a diary, I’d really lose track of these ideas.
Steve, do you have backups in case the server crashes and looses the blog history?
Jason
Dear Mr. McIntyre,
As my comment 43 on the “Contact Steve” page shows, If you were in litigation, either you would get the data and methodology, or the “expert opinions” would be excluded.
Regards,
Will Richardson
#15 “The trouble is that it’s hard to do in 4 GRL pags and I don’t want to spend 2 years in limbo at one of the journals with more space.”
Steve, some journals allow back-to-back publications, especially when the work is considered important. GRL might agree to that if you wrote up your work as two submissions. Send the editor a letter outlining the important consequences of the study. It’s worth asking.
You could alternatively try the same thing at other journals. The two papers often go to a single set of reviewers, who review both papers in tandem. It’s worth exploring this possibility as a way of getting a more complex analysis expeditiously published as two shorter papers.
#17. I can’t imagine that any of these climate scientists has the faintest idea of the type of due diligence that goes on the real world in discoveries. Wouldn’t it be the case that, once the expert opinion was invoked, the invoker would not have the option of withholding the results by withdrawing the opinion? Once it’s been used, they’d have to produce, wouldn’t they?
Steve, you have criticized the prevailing view of statistics in climate science. Don’t you think (since you have been working on this for several years) to present your own reconstruction and explain why it is more accurate than the prevailing view of past climate variability?
I for one would be much more convinced if you could show greater climate variability than any of the dozen or more studies which show that the MWP was warmer than the mean of the 20th century. Yes, it might take you a lot of time and effort, and it is harder than throwing darts, but you might actually come up with something. What you did in MM03 really did not show me anything except that dropping all of the proxies in the pre-1400 period led to anomalous results. Give us a real reconstruction if your arguments are so persuasive.
Ooops, I meant “was not warmer”
JMS
Why do you assume that a reconstruction can be done from tree rings.
This has been discused numerous times before here. Steve has also adreesed this exact point many times.
Of course MBH98 and 99 depended on more than tree rings, as have all of the subsequent *multi* proxy studies.
Besides if he ran his own multi-proxy reconstruction and showed that the temp proxies (all of them: ice cores, sediments, corals and tree rings) had no relationship to temperature — as you argue — don’t you think that might be a big coup? Steve claims to know so much about climate reconstructions, yet the only one he has ever proposed was patently ridiculous. It is time to quit sniping and put up.
Steve: I’ve never "proposed" a reconstruction. This is even acknowledged at realclimate (by Gavin). We stated that Mann could not make his claims about 20th century uniqueness with confidence based on his data and methods and that his claims about his method having statistical skill and being robust to the presence/absence of dendro indicators were false. We’ve shown that you can get very different results with modifications to Mannian methods. This result has been confirmed and generalized by Bürger and Cubasch 2005, who catalogued trivial variations to MBH methods, none of which could be excluded on an a priori basis. I am unaware of any specific claim that we’ve made that has been shown to be "ridiculous". I did a scorecard on our 2003 article last year; maybe I should do a scorecard on our 2005 articles to see how they stand up a year later.
As a matter of interest, (a) why is it that you think that we "proposed" a reconstruction? ( b) what is the basis of your claim that anything that I’ve done has been shown to be "ridiculous"?
Prior to our critical studies on MBH, there had been no critical studies on millennial reconstructions and no one was seemingly even thinking about the prblems. Indeed, von Storch has said that it would have been almost impossible to get them published (even by him) and that we’re owed thanks by the community. Now there’s a small industry – I can identify about 15 published articles on the topic by the VZ group, the Burger, Cubasch group, the Ammann-Mann group and ourselves.
We made constructive suggestions to the NAS panel on directions of research in this area that we thought should be encouraged, and that we thought that they should endorse these directions of research for funding agencies like NSF.
I’ll try to write up a note some time on what I perceive as the role of analytical and critical studies.
JMS, why should Steve produce quickly his own reconstruction if he does not believe that most of the proxies (especially the ones used by a certain team) are not good temperature proxies? In essence, you can get create anything from noise, but what’s the point? Let him get used to the available proxies, let him study the proxies, and maybe one day, we’ll get a real construction that actually tells something about the past temperatures.
I can assure you that it is easy to get a “reconstruction” of any type out of those tree ring proxies available.
For warmer MWP values, see (hot in press):
WeckstràÆà et al: Temperature patterns over the past eight centuries in Northern Fennoscandia inferred from sedimentary diatoms, Quaternary Research (in Press).
re #12: Although I agree a lot with you, I think this blog makes a difference which is not acknowledge by many people: I’m sure some people working in the field are following this blog (they were stupid not to do so), see #13 for a confirmation. So this has to have some impact in the review process. For instance, I can not imagine any honest person reading CA letting through a paper somehow relying on MBH98 attribution statistics although the flaws are only published (so far) here.
re #13: Eduardo, any comment on the Weckstràƒ et al paper (#23)?
Re: #16
I take a backup of the database every few days and I think the ISP does the same. We try to preserve our commentators’ pearls of wisdom for future generations.
JMS,
This is an obvious red herring, and one that’s been tried before. The argument goes like this: If someone publishes a paper in field X using discipline Y, then you can’t criticize the paper if you have not produced work in field X even if you have expertise in discipline Y. What utter nonsense. If Steve has shown us that the leading temperature reconstructions are unreliable at best, then he has performed a great service for us and has advanced science. Yes, you heard that right: Advanced Science. Science can’t advance unless the incorrect theories and bad data are weeded out, otherwise we’d still think the moon was made of cheese and that the sun revolved around the Earth. Steve is doing good work here and does not need to publish a reconstruction to be accepted by the current in-crowd in the paleoclimate club.
Re: #16 and #27.
Major search engines cache most everything after a day or two so even if the ISP goes poof in the night, Google has copies. Here is MSN’s curent cache of ClimateAudit’s top page.
Decades from now, historians’ best friends will be Google, Yahoo and MSN Search.
Re: 20.
Taking your acknowledgement as a starting point, MM03 undermined the claims in MBH98 that the MWP did not exist as a global phenomenon and that the present temperature was ‘extraordinary’. These claims have subsequently been supported by reconstructions such as Esper and Moberg, which show a much greater variability than MBH98. So your requirement for Steve to produce a reconstruction with greater variability is moot – it has already been done! When are you going to stop pretending there is a consensus ‘prevailing view’?
Here is something from my “field of dreams.” Namely, it is the day when the most serious of the world’s serious signal analysts (from Applied Maths, Defences, Avionics, Telecoms, etc) enter into a substantial conclave with the world’s so called “Climate Scientists.” Imagine it. One cabal, well imbued with poles and zeros, overshoots, filter theory, stability analysis, Fourier Transforms, initial conditions, signal to noise ratios and the like, coming to grips with splices as noted above. A serious, no holds barred, critical review of what is right and what is not right about all that has been done with climate history assumptions and climate prediction. People who have no dog in the hunt, embarking on their first hunting season. Ah yes, imagine it!
I begin studying principal components tonight, Steve S., as a background to my eventual dissertation, which will likely be in the neighborhood of independent component analysis and blind-source methods. Signal analysts indeed!
FWIW, my interest in PCA w.r.t. climate science is actually independent of the need to learn it as a background for the dissertation. Blind source methods are the topic du-jour in the signal processing community and the best avenue for finding untouched research opportunities.
Mark
re #31: New lyrics for “Imagine” by John Lennon? 😉
re #32: With that in mind, I suggest that you study PCA from Hyvàƒ⣲inen et al: “Independent Component Analysis” (Chapter 6, check the references therein if you need more details). Sorry, but they don’t cover Mannian PCA 😉
RE” #33. You might say that I’m a dreamer, but I’m not the only one … 🙂
Re: Comment 19
Fortunately, in court, the trial judge is the gatekeeper of what evidence the jury is allowed to hear. If an “Expert” is offerred as a witness, he is required to disclose not only the data used in making his conclusions, but also any data he considered and chose not to use. He must also explain his data selection method, his methodology, and formulas, processes and protocols. If he uses software in reaching his opinion, he must produce a working copy of the software to his opponents. If the “Expert” refuses to produce any one of those items, cannot produce each those items, or cannot or will not adequately explain his methodology, the jury never gets to hear the “Expert’s” opinion.
Upon occasion, an “Expert” will inadvertently or purposefully withhold data or inadequately disclose his methodology and is as a consequence allowed to offer his opinion to a jury. If it is later proved to the trial judge that the “Expert” withheld data or inadequately disclosed his methodology, even inadvertently, the trial judge will then strike the “Expert” opinion and instruct the jury to disregard the testimony.
I noticed that several States have sued the Federal Government in a effort to force the EPA to take action restricting greenhouse gas emissions to prevent AGM. Perhaps they will list Mann as a “Expert” witness and he will finally be forced to fully disclose his data and methodology.
Re #35, Will Richardson
I’d love to see Mann in the witness chair with a well-briefed barrister nailing him down. However, I don’t imagine he would come there voluntarily.
Is there any basis on which a private citizen could bring suit against a state government – say, someone whose business has been damaged by CO2 emission restraints – and summons Mann ?
Jean S… the book I have is Adaptive Blind Signal and Image Processing by Andrzej Cichocki, Shun-ichi Amari…
Not too bad, but there are still some grammar errors (only 2nd ed.) and no problems at the end of each chapter. Chapter 3 is PCA, which I start tonight.
I’ll look into your recommended text, thanks.
Mark
Oh, curiously, I just looked at your reference for Hyvarinen’s book and I see it is co-authored by Juha Karhunen… any relation to the Karhunen of Karhunen-Loeve fame? My advisor seems to think the latter is probably deceased, but maybe an offspring?
mark
Re #36: “Is there any basis on which a private citizen could bring suit against a state government – say, someone whose business has been damaged by CO2 emission restraints – and summons Mann ?”
A number of lawsuits have been filed by proponents of AGW to force the Federal Government to impose CO2 emission controls. The City of Boulder has filed suit. Calling Mann or any other scientist, both believer or skeptic, would be an option.
I don’t know the status of these suits but the AGW community should be careful what they wish for. The skeptics will be able to present their entire case under oath and on the record. To quote President Bush, “Bring it on!”.
re #37: The approach in those two books is rather different, and which one is better for you depends from your background/preferences. In general, IMO, C&A is more “technical” (and contains more information) but HKO is much easier to read. Addition to these two books there is a newer book on ICA by James Stone. I have not seen it, but judging from the Table of Contents (and Amazon reviews) seems to be a rather good book on the topic. BTW, if you really want to learn the statistical fundamentals of ICA/BSS, I suggest you to read/master the following two papers:
1) P. Comon, Independent Component Analysis, a new concept ?, Signal Processing, 36(3):287–314, April 1994.
2) J-F. Cardoso, Blind signal separation: statistical principles, Proceedings of the IEEE, 90(8):2009–2026, October 1998.
re #38: Juha Karhunen has no relation to Karhunen-Loeve, and to my knowledge is not an offspring either.
C&A is EXTREMELY technical. The only background they give, btw, is methods for solving linear systems of equations with unknown sources (and known mixing matrix and observation vector). I’ve had plenty of instruction in that realm, but I went through it just to have a good read on the methods they prefer since those methods will appear in their later solutions (I’m guessing).
I saw the Stone book listed on Amazon, and may buy it at some point, maybe even the HKO book. I have access to both of those papers via IEEE as well. Thank you for the pointers. Odd that the new Karhunen would be unrelated, yet doing nearly identical work with the same name…
In the end, this all has to result in a dissertation topic, and, preferably, an actual dissertation! 🙂
Mark
I would of liked to here more about this stabilizing method. A link, a justification some mathematics something.
? Stabilizing method?
Mark
#43
“Figure 2. Top – archived version. Middle – 3 series version pre-stabilization. Bottom – 3-series version, post-stabilization.”
Ah, gotcha. Not sure, actually, though on inspection, I would guesstimate a moving average filter which seems to get discussed often around here. 🙂
Mark
#45 It is too bad that the hockey team doesn’t realize that a moving average filter is exactly what you don’t want. If anything you want a high pass filter rather then a low pas filter as there is simply not enough data to identify low frequency signals especially since the number of parameters you are trying to fit is large.
Perhaps we should weight our information in the frequency components as:
1/sqrt(f*T) Where T is the data length in time and f is the frequency. From this frequency weighting we should be able to devise a whitening filter.
The signal processing is coming back to me. So recall that the noise decreases with 1/sqrt(n) the number of independent measurements. Since a sign wave takes a whole period to repeat, each period constitutes an independent measurement. Thus the amplitude (stand deviation) of the noise should fall with one over the square root of the number of periods. That is 1/sqrt(f*T) where f is the frequency and T is the data length.
The power spectrum is the magnitude squared of the frequency distribution. That is:
Ps=Ys Ys*
Where Ps is the power spectrum, Ys is the frequency spectrum and Ys* is the complex conjugate of the frequency spectrum. Taking the inverse Fourier transform of the power spectrum we get the autocorrelation function.
Rs(Tau)=F^-1{Ps} where Rs is the auto correlation function Tau is the time delay
F^-1 is the inverse Fourier transform
The correlation matrix can be written from the auto correlation function as follows:
RR=
[Rs(t1) Rs(t2) … Rs(Tn)]
[Rs(t2) Rs(t1) … Rs(Tn-1)]
…
[[Rs(tn) Rs(t1) … Rs(T1})]
Now that we have the correlation matrix we can whiten the linear equation
Y=Ax
By multiplying both sides of the equation by the square root of the inverse of the correlation matrix. The inverse of the correlation matrix is known as the information matrix.