Climate Audit

Fred

chuggachuggachuggachuggachuggachuggachuggachuggachuggachuggachuggachugga

The sound my washing machine makes in it Spin Cycle.

Jonathan Dumas

To my eye, the yellow line indicates ~1920 rather than 196o.

Steve: Corrected.

stevemcintyre

For those who say that blog posts aren’t reviewed: I mislocated the yellow line initially. Within 10 minutes, 4 different readers reported the error.

Jack

What is the caption for this figure? It’s hard to determine if the figure is misleading when the caption that was included with it is not presented as well.

Jean S

Jack, although I do not understand how any caption would change the situation, here is the link to the original WMO statement:

Click to access wmo913.pdf

stevemcintyre

I’ve added in the caption and link. Thanks.

Jack

Jean, thank you for the link. It would make a difference if it were clear from the caption what the figure was comprised of. Disappointingly, it’s not clear from the caption so I agree it’s not a good figure. I don’t agree that the figure is intentionally “hiding” anything, but you must be clear about what data you are using and how you are using it.

stevemcintyre

Jack, have you read the emails in the two months leading up to the “trick” email?

Billthetechnicalwriter

Let me understand: The tree-ring data post 1960 does NOT represent Temperature, however, the tree-ring data up to 1960 supposedly DOES represent Temperature. Maybe the answer to the Divergence “problem” is obviously “hidden in plain sight” – that is: NONE OF THE TREE RING DATA REPRESENT TEMPERATURE DATA at all. Sorry, paleodendroclimatologists – it is time for you to select an honest branch of science in which to end your careers (snip if desired).

Dirk

I can’t believe it’s not universally accepted that CRU has BAD SCIENCE. Because the scientific method means presenting your data and methods such that it can be replicated. If you submit a Chem 101 lab without data and calculations, you flunk. How anyone can submit climate “findings” without data and calculations to allow others to go where they went and pull cores like they pulled and get equal results is beyond me. If you can’t replicate it, you can’t consider it real.

When I was an engineer in industry, I qualified materials for use, often seeking cost reduction. I remember a couple times that the data did not show equivalent reliability- usually not all the samples were
“bad”, just a few- so we couldn’t use the material until understanding why the “bad” data points were “bad”. We couldn’t just hide them.

Thank goodness these guys are working on the weather and not food additives or building materials.

Bill Jamison

Did anyone else see the report in the Daily Telegraph that claims CRU has decided to make their data publicly available?

Looks like a victory for Steve and everyone else that has been trying to get the data from CRU!

http://www.telegraph.co.uk/earth/copenhagen-climate-change-confe/6678469/Climategate-University-of-East-Anglia-U-turn-in-climate-change-row.html

David

It looks like a comment I posted earlier in this thread has been skipped over and is still awaiting moderation.

Pistolus

Mann looks like he is ready to throw Jones under the bus, from his missive to reporters and others defending himself (and only himself) Mann writes the following (reproduced by Joe Romm):

“3. “Can you delete any emails you may have had with Keith re AR4? Keith will do likewise. He’s not in at the moment -minor family crisis. Can you also email Gene and get him to do the same? I don’t have his new email address. We will be getting Caspar to do likewise.” (from Phil Jones)

This was simply an email that was sent to me, and can in no way be taken to indicate approval of, let alone compliance with, the request. I did not delete any such email correspondences.”

http://climateprogress.org/2009/11/28/climategate-michael-mann-hockey-stick-copenhagen-diagnosis/#more-14670

He’s got some other lengthy explanations there also.

Oldjim

Jackpot!!
http://www.telegraph.co.uk/earth/copenhagen-climate-change-confe/6678469/Climategate-University-of-East-Anglia-U-turn-in-climate-change-row.html
Leading British scientists at the University of East Anglia, who were accused of manipulating climate change data – dubbed Climategate – have agreed to publish their figures in full.

Robert Wood

http://www.telegraph.co.uk/earth/copenhagen-climate-change-confe/6678469/Climategate-University-of-East-Anglia-U-turn-in-climate-change-row.html

So, UEA and The Had Crew have decided to publish that which the dog ate?

On another note, I see some MSM outlets recognizing Climategate, but do not understand or report the significance of all the other documents, software and data files.

docmartyn

Steve, have you any idea what the correlation between the tree ring data used and the independently measured temperature, THAT WAS USED IN THE CALIBRATION PERIOD.

This is the bit that interests me. What is the R2 for tree rings vs temp, in the bit history they claim it was correlated. They are claiming errors of +/- in the derived data. This would mean that the correlation needs to be very good indeed <0.5/20.

Jonathan Fischoff

Explanations like the ones they are pitching only work for the ignorant. Unfortunately for the cause, they are motivating intelligent informed people to be skeptical.

marie elks

Untitled Document

Mr Steve–
Have you looked at the file SOAP-D15-berlin-d15-jj.doc in the documents folder
of the .zip file? Does this sound even reasonably sane? To an average person
it sounds quite insane.

At the same time, a series of millennial climate simulations was conducted
using estimates of solar irradiance (S), greenhouse gases (G), and volcanic
aerosols (V) as external historical forcing. Ideally, the internal (=natural)
fluctuations (the “noise”) as well as the forcing response (the
“signal”) is of the same scale (and shape) in all models and
equals reality. And with a sufficiently large signal-to-noise ratio
(SNR) for a given forcing it should be possible to distinguish the signal
from the noise and determine the sensitivity of the climate system to that
forcing; moreover, proxies and models would show the same signal. In the
real world, however, both noise and signal tend to depend on the specific
model, and since proxy reconstructions are imperfect as well the real signal
is deeply obscured.
[image]

Fig. 2 shows two millennial simulations using all three forcings (S, G,
and V), one from the Hadley centre model HadCM3 (“all” simulation)
and the other from the DKRZ model ECHO-G (“Erik” simulation).
Obviously both models deviate considerably from each other and from
the instrumental record (Jones et al., 1999), with a cold bias of as much
as 1K (ECHO-G) and 1.5K (HadCM3). This is compensated for by a tropical
and SH warm bias (not shown) which gives a much better fit for the global
average. At least some of the Northern Hemisphere difference between
the two simulations may be because the HadCM3 all simulation has aerosol
forcing, whereas the Erik simulation does not (Note that GCM parameters
are tuned using that average, among others, as a target.) It is unlikely
that from those raw time series a forcing signal can be detected.

[image]
However, under rather mild assumptions it can be assumed that the
response of the system will not change drastically if transfered from the
too cold to the normal state. Therefore, it is justified to replace both
simulations with a corresponding best estimate to the CRU data, just like
the proxies in Fig. 1.

curious

Oldjim – not so sure about the “Jackpot!!”. IMO there are two lemons on the line at the moment and the third is maybe within nudging distance. Next thing is: What is inside the machine to payout?…

docmartyn

Malcolm Hughes- I am confident that, before AD1850, (stripbark’s) do contain a record of decadal-scale growth season temperature variability. I am equally confident that, after that date, they are recording something else.

Alleged CRU Email – 969618170.txt

Quoting tom crowley :

Dear Malcolm and Keith, as I discuss in my Ambio paper the “anomalous” late 19th century warming also occurs in a LaMarche tree ring record from central Colorado, the
Urals record of Briffa, and the east China phenological temperature record of Zhu.

Alpine glaciers also started to retreat in many regions around 1850,
with 1/3 to 1/2 of their full retreat occurring before the warming that
commenced about 1920.

The Overpeck et al Arctic synthesis also discusses warming before 1920 -that record matches very closely the Mann et al reconstruction in other details back to 1600.
Unpublished work by us on coral trends also suggests slight warming between about 1850-1920.

So, are you sure that some CO2 fertilization is responsible for this?
May we not actually be seeing a warming?

Tom
– – – – – – – – – — – — – – – – –
Thomas J. Crowley From: mhughes@xxxxxxxxx.xxx
To: tom crowley
Subject: Re: old stuff
Date: Fri, 22 Sep 2000 06:22:50 -0700
Cc:

Dear Tom,
The difference between the Campito Mountain record and, for example, the one
from the Polar Urals that you mention, is that there is no meaningful
correlation between the Campito record and local temperature, whereas there is a
strong correlation in the Polar Urals case. I give references to the work
reporting this phenomenon at the end of this message, but I’m afraid I’m missing
the references to the technical comments that are being responded to in the last
two. If you examine my Fig 1 closely you will see that the Campito record and
Keith’s reconstruction from wood density are extraordinarily similar until 1850.
After that they differ not only in the lack of long-term trend in Keith’s
record, but in every other respect – the decadal-scale correlation breaks down.
I tried to imply in my e-mail, but will now say it directly, that although a
direct carbon dioxide effect is still the best candidate to explain this effect,
it is far from proven. In any case, the relevant point is that there is no
meaningful correlation with local temperature. Not all high-elevation tree-ring
records from the West that might reflect temperature show this upward trend. It
is only clear in the driest parts (western) of the region (the Great Basin),
above about 3150 meters elevation, in trees old enough (>~800 years) to have
lost most of their bark – ‘stripbark’ trees. As luck would have it, these are
precisely the trees that give the chance to build temperature records for most
of the Holocene. I am confident that, before AD1850, they do contain a record of
decadal-scale growth season temperature variability. I am equally confident
that, after that date, they are recording something else.
I’m split between Harvard Forest and UMASS these days, and my copy of your paper
is not with me today. I’d be interested to know what the name of the site for
the LaMarche central Colorado record was.
Cheers, Malcolm

DeputyHeadmistress

A few hours after promising to release the data, according to the UK Sunday Times, the university was admitting that all the raw data from the 80s and before had been dumped when they moved locations:

““SCIENTISTS at the University of East Anglia (UEA) have admitted throwing away much of the raw temperature data on which their predictions of global warming are based.

It means that other academics are not able to check basic calculations said to show a long-term rise in temperature over the past 150 years.

The UEA’s Climatic Research Unit (CRU) was forced to reveal the loss following requests for the data under Freedom of Information legislation. “

The data were gathered from weather stations around the world and then adjusted to take account of variables in the way they were collected. The revised figures were kept, but the originals — stored on paper and magnetic tape — were dumped to save space when the CRU moved to a new building.””

It’s okay though, because they have the ‘value added, homogenised and quality controlled data.’ And they say so, thus it must be true, yes?

Ian

Ok, that didn’t work either…it looks right when I put it in, but it loses the signs…

Without trying to prejudice this work, but also because of what I
almost think I know to be the case, the results of this study will
show that we can probably say a fair bit about [less than] 100 year
extra-tropical NH temperature variability (at least as far as we
believe the proxy estimates), but honestly know fuck-all about what
the [greater than] 100 year variability was like with any certainty (i.e. we know
with certainty that we know fuck-all).

Andrew

All the excitement regarding the forthcoming data will (I predict) end up as nothing more than a (hopeful on the CRU part) Red Herring. Unless they are will ing to give up all the Raw data, method, etc. it amounts to a hill of nothing. Again it is most important to ask now, through what ever means possible, for this. CRU will likely give the data that makes up their graphs and thus point to the releasing of the data as as means to shut up the skeptics. “There. We have given you the data….now go away…”. I’m sure Steve M will be in contact with CRU to discuss what is coming….well I hope so. Steve have they been in contact? You with them?

effinayright

Can someone help a poor layman out here? When I was a kid in college an elec. engineering prof used to have us do quick problems in class. Once , when a student gave “the answer” as “6.346 Ohms”, the prof said “Say, that’s some slide rule you got there”. The problem, of course, was the precision that could be derived from a slide rule.

So I’m asking how all this world-wide data supposedly “measured” over a thousand of years or more can conveniently be precise to the tenth of a degree, and “smoothed” at that.

I’m told that areas of the earth for which there were no temperatures collected were assigned values . Well, if so, how could anyone assign a value, to a tenth of a degree? How can tree-ring proxies, given the variations in rings due to soil characteristics, water, humidity, cloudy-vs-sunny weather, etc. be measured with such precision?

Can someone point me (and others ) to a site or article explaining how such precision is achieved, given the imprecision (or downright inaccuracy) of early temp measurements, directly or through proxies?

Or is that just one of the issues McIntyre et al have been raising?

DWhite

effinayright said:
[i]How can tree-ring proxies, given the variations in rings due to soil characteristics, water, humidity, cloudy-vs-sunny weather, etc. be measured with such precision?[/i]

Hey, it’s Climate Science. If you like the one about 0.1 degree precision, have you heard the one about “teleconnection”? That’s the one where your tree, say in the western US, doesn’t respond to the local climate surrounding that tree, but does respond to the globally averaged temperature. No, I’m not making that up.

I think there’s really only one way an organization can get away with such tripe, but that gets us into politics, which are off limits here. The whole thing is really quite breathtaking.

OzzieAardvark

snip – politics

OzzieAardvark

Uh… Interstate 44. Sorry 🙂

vg

might this help?
#

This story from the London Times forthrightly states that the raw data that Phil Jones and his team depends upon has been lost.
http://www.timesonline.co.uk/tol/news/environment/article6936328.ece
Can you clarify whether this is true or not.
Thanks

[Response: No. The original data is curated at the met services where it originated. – gavin]

Comment by Bernie — 28 November 2009 @ 11:34 PM

galileonardo

I just posted this over on WUWT, but on the subject of the lost raw data, have no fear. Jones wrote the following in his 1255298593.txt email to Rick Piltz:

“The original raw data are not lost either. I could reconstruct what we had from some DoE reports we published in the mid-1980s. I would start with the GHCN data. I know that the effort would be a complete wate of time though. I may get around to it some time. As you’ve said, the documentation of what we’ve done is all in the literature.”

Sounds like he may be forced to waste some time after all.

Shona

According to Courtillot, tree, not rings, but density are extremely good proxies for temp when temp is changing rapidly, however when temp plateaus, then trees adapt and they no longer show temp well. (As someone who has grown plants for a long time this “sounds” right, in normal conditions they aren’t too sensitive to temp, but react strongly when temp changing significantly.)

I’ll look for the place in his Nantes lecture where he shows the graphs.

Steve I seem to be caught in the spam trap

Lucy Skywalker

Please delete my OTT comments at the end of some of my lines. Apologies.

billcat

Somewhat OT, but Rasmus Benestad is at it again in the Norwegian research online: forskning.no
http://www.forskning.no/artikler/2009/november/235924

I ran the thing through google translate and tidied up afterwards. Somewhat, his original Norwegian prose itself look like it is the result of a google translate effort. :-s

A lesson from the CRU hack?

One should certainly be carful of what you write in e-mail to colleagues. For perhaps we really are living in 1984-the world (ie 1984 from George Orwell) in the workplace, when confidential e-mails so easily can be posted completely open?

Even if they are supposedly be job related, the human factor also matter when you communicate with each other. Or perhaps one should always write in a stiff and formal style to all his colleagues? But who does that today?

Not climate scientists anyway -as we have just learned, and it is quite clear that someone attempts to create a sensation out of it.

Because some of the things that came out was certainly ugly and embarrassing. But the important point is that it was only an internal dialogue between colleagues who did not involve any outrageous actions. This is of course possible to check. The IPCC report contains the answer – which articles are missing which shouldn’t be included? [Comment: Yes, this is a direct translation] If so, should we not have heard of this before now?

Just talk

The analogy is two friends talking about a bank robbery, and then they are overheard and there is a big commotion. But there has not been a bank robbery, and while they only talk about it, they have not done anything illegal (even if such talk is probably not wise). Similarly, the e-mails contained a lot of talk, but no unethical action. The reason is that there are many scientific journals (I haven’t got the overview) and many scientists at a high level are not the best of friends. There are more analysis of the global mean temperature, in addition to sea-level temperatures, from other sources.

But I also certainly had a strange feeling when I had finished reading the book ‘The Republican War on Science’ (Chris Mooney), right after the hack at the Climate Research Unit (CRU, University of East Anglia, England). In the book, Mooney discusses the climate debate along with a number of other issues such as environment contaminants, endangered species, diet-sugar [sic], intelligent design, stem cells, abortion, tobacco, and sex education. In all these cases, he finds a recurrent pattern of contempt towards knowledge and research.

The theme of the book suddenly came much closer to home- because it feels so much more real after I myself was spun into the drama surrounding the hack at CRU. One of my emails was also published, as I had to explain to a couple of journalists. Moreover, RealClimate.org had been tried hacked in connection with this raid.

The most obvious similarity between the CRU-break and The Republican War on Science touches legislation such as Freedom of Information Act ‘, the Shellby amendment, and the Data Quality Act, which among other things, provides industry and interest groups free access to all public data. It is a good principle, but in the wake of these there has also have been some unfortunate experiences with unreasonable arguments based on these data. Confusion is created when matter is technical, difficult, complicated and hard to get an overview of.

Out of proportions

We have seen ugly concrete examples that this has happened: namely, “Hockey Stick graph” of Michael Mann. This has also probably had an impact on some climate researchers’ behavior, and created a resistance among some to give out data (another aspect of the matter is that more data are commercial, and can not be given out freely to a third party). Moreover, strong personal conflict also affected the situation, and this is obviously not good.

What also emerges is an extremely caution expressed in the e-mails. The aftermath of this CRU-break has also to the full shown that this has been prudent. Because some climate skeptics have turned an illegal hack on its head, and tried to make the victims emerge as the villains. Climate scientists are rendered suspect and their opponents blow things far out of proportion.

When one reads the responses to Bjørnar Kjenslis reportage of the case on forskning.no (No climate conspiracy in sight) [Comment: A whitewash, nothing-to-see-here article at forskning.no, in which the journalist was skewered in the comments], it is almost frightening how unreasonable reactions on this. These show a blind witch hunt of climate scientists.

Simply associating the break with Watergate scandal, shows how the case is turned upside down. Because it is not illegal to send e-mails to colleagues, and it was a burglary at the CRU which was illegal. Nixon however did something illegal, while the Watergate investigation in itself was not.

Calling the situation the ‘Watergate of Climatology’ – or ‘Climate Gate’ – is also a trick that could well have been taken directly from ‘The Republican War on Science’. That there is a conspiracy when climate scientists communicate with each other is absurd – even if some scientists send some naive, unethical, or poorly thought out emails.

Release of data

It can nevertheless not be justified to not release the data. It is also important to build up knowledge in the poulance so that they understand what the data means and their limitations. This should be a job for us climate scientists, but the problem is that there are not very many climate scientists, and we don’t have such large resources.

I do not think we have a good enough capacity to explain or guide if there is a high demand. Often there are people who misunderstand the data and we must correct the abuse, especially when used in a political context by parties that have the public relations expertise that only wealthy organizations can buy. The fact that you could buy T-shirts printed to ‘Hide the Decline’ – ie a reference to the hack – less than 3 days after it became known, shows how well-organized adversary climate scientists are meeting. And the short film ‘Hide the Decline’ is posted on YouTube less than 7 days after the hack was known.

Chris Mooney writes that when the Office of Technology Assessment (OTA) was closed, the stage was set for the so-called “science courts” in the U.S. Congress. It became a play to the gallery event that gave the wrong impression of academic expertise and where the jury had no basis on which to consider the substance. But this is also apparently come to Norway, where some Norwegian political parties seem to have copied the American model.

Nina Kristiansen[Editor at forskning.no] writes in the commentary ”stolen insight into climate research” that climate research should open all doors, also in the room where uncertainty and disagreement prevails. And I share her views on this, and she also has many points. But I think that there are also several other issues that should be highlighted. Not least, issues surrounding climate skeptics. Because, as Chris Mooney explains, there is a lot of shady politics going on in the framework and guidelines around the science. We also need much greater transparency in all the parties and the landscape around this, especially around the media, PR industry and policies that are affected.

Media does not do its job

The media is important because they have much power and affect so many people. Here the resources should be used properly, and therefore transparency is essential. Some examples are why is it is that debate programs on TV about climate often do not take in the scientists who actually researching the topic, but are good at grabbing in profiled skeptics. Another example, we took up in a letter to the editor to Morgenbladet “Forskning? No!”, and we have argued the case for more openness about editorial decisions in among others forskning.no.

To me it seems that the media has not done its job. Some of us have argued for this in the Tech Weekly ( ‘Where is the critical and burrowing journalism?’, TU 4109). There are few journalists who really have an overview of people and backgrounds, and Norwegian journalists is a very rare sight on major scientific conferences. They rather prioritize sporting events.

But I hope that forskning.no shows up on the next European Meteorological Societys annual meeting in Zurich in September 2010. There is a media & communication session there which should attract.

Mina

Steve
I think you will find the problem is your lacking grasp of statistics. Norwegian climate researcher Rasmus Benestad explains on ‘forskning.no'(http://www.forskning.no/artikler/2009/november/235924), a website for science news, that you do not know what you are talking about. This is Benestad’s reply to a comment about the hockey stick debate (translation by me):

“I can explain why I am not convinced by Ross McKitrick or Steve McIntyre. Three episodes stand out:

1. McIntyre has used a lot of energy to argue that the curve’s “hockey stick” shape is suspect and that it is a result of principal component analysis (PCA; which is a piece of linear algebra often used in data analysis). This question is way off because PCA does not change the information of the data, but reorganizes them in a way that makes them easier to handle. It would on the other hand be reasonable to ask how much of the variance the different modes explain, and how much weight they are given in the following regression analysis. So McIntyre’s statement indicate that he really didn’t know what he was talking about (another thing is that Mann did the analysis again without involving PCA and got equal results).

2. McKitric [sic]& Michaels argued in an article in 2004 that most of the global warming could be explained by economic activity (http://www.realclimate.org/index.php/archives/2004/12/are-temperature-trends-affected-by-economic-activity/). But they had made some fundamental errors which made their conclusions invalid – they had assumed that all measures and economic data were totally independent of each other. It is one thing that the same economic factors are valid for all measures from the same countries, but you must also account for the correlation in space.

3. McIntyre referred to a statistical test I have developed (published in 4 peer reviewed magazines, freely accessible as open source on http://cran.r-project.org – iid.test). His comment was complete nonsense [he uses a derogatory Norwegian expression that has no English counterpart, and is used when someone gives a nonsensical answer that has nothing to do with the question].

I believe that neither McKitric[sic] or McIntyre (or Michaels) have a very good grasp of statistics – but that doesn’t stop them from being very loud. Unfortunately my impression is that much too few researches have a very good grasp of statistics.”

So there.

Steve: As to 1) our criticisms of Mann’s PCA were acknowledged by both Wegman and North reports. Contrary to Benestad’s false claim, we identified the effect of the PCA modes and showed that MAnn’s hockey stick derived from strip bark bristlecones, proxies known to be questionable as “temperature” proxies and which the NAS panel said should be “avoided”. There are other ways of using the flawed bristlecones besides Mann’s flawed PCA and these have been discussed and criticized on many occasions at CA. 2) I had no involvement with the Michaels paper and do not understand why any merit or lack of merit of that paper should reflect on me. 3) Could you refer me to the comment being criticized? For example, I agree with Rasmus on the following prediction made at realclimate, but do not regard it as a particularly searching observation:

I can easily predict that the summar will be warmer than current (winter) conditions…

William Porter

snip: the C-word and e-word are strictly offlimits at this blog

stevemcintyre

New readers, please do not post comments from news articles on a thread that has a technical point. It clutters the thread.

Alexander Harvey

Could someone please explain something to me, very slowly. Where am I going wrong?

There are IPCC and WMO graphics that use a Briffa tree ring series.

In the make up of the IPCC graphic:
An original series existed that ran till 1994.
If a 50 year (-25/+25) smooth was used, this would be good until 1994-25 = 1969.
It was decided not to include the series in the grapphic post 1960.
But a truncated version of a smooth which would be good until 1969 (50 yr) was not used.

It was the data that was truncated at 1960 (A 50 year smooth now good until 1935).
A new smooth was produced upto 1960 using 25 years of padding data (50 yr smooth).

At some point someone knew what both the original data post 1960 were and the padded data were.
At that point it would have been clear precisely how misleading the final smooth was (the difference between the original series post 1960 and the padding data, or the difference between the resultant smooths).
If that be the case it is quite clearly misleading, and if some one person knew all the stages I think they be less than honest brokers.

In the make up of the WMO graphic:
I simply do not know what has gone on. All the smoooths seem to run upto 2000, so they are all passed their good till dates. All the references are to papers published pre 2000 so there is a lot of data added to the end of all the original series to create the smooths.
In the case of Briffa this would require extending from 1994 till 2025 (31 years out of 50).
But I think I am right in thinking that it be the truncated data that was actually used, and it was padded from 1960 till 2025 (65 years out of 50). If my thinking be right that does not make any sense at all. The year 2000 would be 15 years after the point were the smooth became 100% fictitious. The last years of the other smooths are more than 50% fictitious.
If that be the case then hese are not smooths as I know them, these I would know as extrapolations.

So which bits have Igot are wrong? I am not being rhetorical.

Alex

I am genuinely baffle, and I expect bamboozled.

Alex

EdeF

I would think a simpler way to smooth this data would be to take a
Fast Fourier Transform of the time series, convert it to the frequency
domain, cut out the high frequency components and then transform it
back to the time domain.

Alexander Harvey

EdeF,

I really do not think that FFT will address the question of what to do about the end points. The data is not cyclic, so what do you do? You could use a version that only integrates on a domain restricted to the data period but then the sinusoids would loose their orthogonality and no unique transform would exist.

As far as I am aware the only fair ways of smoothing require you to lose a period of at least one half of the smoothing period at each end. By “fair” I mean ones that do not introduce arbitary additional data or vary the length of the smoothing period.

I think it is the question of what to do with the end points that is the basis of all the issues here and I do not see how using FFT would alter that.

Alex

mikep

Mina,
It’s a bit difficult to understand Benestad’s criticisms from your description, but the link to real climate goes to a post which itself sems to display misunderstanding. One criticism of the McKitrick Michaels work seems to have been whether spatial autocorrelation was properly treated. But rela cliamte seemed to get confused. The dependent variable (temperature by grid) did indeed show spatial autocorelation. But the residuals of the equation were white noise, and that is what matters. There is plentyof work on this at McKitrick’s site
http://ross.mckitrick.googlepages.com/
as well as some CA postings, which I do not have directly to hand.

PaulM

The graphs posted at UEA seem to do nothing for them except expose the “hide the decline” trick. Their rambling, desperate comments are a PR disaster.

Very good coverage in the Sunday Times today. Two articles by their enviro correspondent, one about the admission of raw data and one about the whole email affair, and another article about how CRU need to engage with critics.
The BBC is still in denial.

Some are saying its good that they are saying that they will release the data. But theyve been promising that for years, right?

ThinkingScientist

I agree with the comments by Alexander Harvey. If you are applying a centred smoother than the last valid point is the last year in which the full filter is applied. You cannot know what a centred 50 year average will look like until you have a further 25 years of data. Any attempt to do so is arbitrary and misleading.

If you want to try and run the filter to the end you might taper the filter ie progressively shorten the filter length as you approach the end of the series. However this then means that the ends of the time series change from being a low frequency estimate to a broad band estimate. Still misleading.

Padding the end of the time series with any particular value is arbitrary and essentially changes the stationarity assumption. Of course if you want to get a particular result then end padding with a “preferred value” is the method of choice.

On the question of filtering in the frequency domain (via FFT) rather than in the time domain, you still have problems. If you frequency domain filter than the trauncation in the frequency (or slope, if its not too severe) will create a “ringing” effect when you back transform to the time domain. This can be particularly noticeable if you have a large transient (eg 1998 el Nino) near the end of the series.

All methods of filtering suffer from side effects at the end of the series. You can’t know something that hasn’t happened yet, no matter how hard you wish it.