“A good way to deal with a problem”

In the UEA statement of Nov 24, 2009, Phil Jones said:

CRU has not sought to hide the decline. Indeed, CRU has published a number of articles that both illustrate, and discuss the implications of, this recent tree-ring decline, including the article that is listed in the legend of the WMO Statement figure. It is because of this trend in these tree-ring data that we know does not represent temperature change that I only show this series up to 1960 in the WMO Statement.

Here is the corresponding figure in the UEA statement. I’ve added a yellow line for 1960. Although Jones’ statement says that he “only showed this series up to 1960″, the series attributed to “Briffa (1999)” obviously continues past 1960 into the 1990s. Jones said that CRU did not attempt to hide the decline in the Briffa reconstruction. And it is true that earlier articles did not take advantage of “Mike’s trick”. However, although the “real” Briffa reconstruction goes down after 1960, the series in the diagram attributed to “Briffa (1999)” goes up. The decline in the Briffa reconstruction is not shown; it is hidden. Gavin Schmidt of real climate says that this is “a good way to deal with a problem”. I disagree (and recorded this disagreement in a related context in connection with IPCC AR4 as discussed elsewhere.)


WMO Original caption: Northern Hemisphere temperatures were reconstructed for the past 1000 years (up to 1999) using palaeoclimatic records (tree rings, corals, ice cores, lake sediments, etc.), along with historical and long instrumental records. The data are shown as 50-year smoothed differences from the 1961–1990 normal. Uncertainties are greater in the early part of the millennium (see page 4 for further information). For more details, readers are referred to the PAGES newsletter (Vol. 7, No. 1: March 1999, also available at http://www.pages.unibe.ch) and the National Geophysical Data Center (http://www.ngdc.noaa.gov). (Sources of data: P.D. Jones, K.R. Briffa and T.J. Osborn, University of East Anglia, UK; M.E. Mann, University of Virginia, USA; R.S. Bradley, University of Massachusetts, USA; M.K. Hughes, University of Arizona, USA; and the Hadley Centre, The Met. Office).

84 Comments

  1. Lorne LeClerc
    Posted Nov 28, 2009 at 3:44 PM | Permalink

    Hello Steve, it looks like your yellow line lies at around 1916 – not 1960 as stated.

    Take care,
    Lorne

    Steve:
    corrected.

  2. David44
    Posted Nov 28, 2009 at 3:44 PM | Permalink

    Steve –

    Maybe I’m misreading the time scale, but your yellow line looks a good bit earlier than 1960, maybe 1920?

    p.s. Thanks for all your hard work, expertise and backbone.

    David

    Steve:
    Corrected.

  3. Fred
    Posted Nov 28, 2009 at 3:45 PM | Permalink

    chuggachuggachuggachuggachuggachuggachuggachuggachuggachuggachuggachugga

    The sound my washing machine makes in it Spin Cycle.

  4. Gary Anderson
    Posted Nov 28, 2009 at 3:47 PM | Permalink

    The yellow line seems to be at 1910. Shouldn’t it be to the right of the last hash mark?

    Steve: My bad. I’ll re-post.

  5. Jonathan Dumas
    Posted Nov 28, 2009 at 3:51 PM | Permalink

    To my eye, the yellow line indicates ~1920 rather than 196o.

    Steve:
    Corrected.

  6. Jonathan Dumas
    Posted Nov 28, 2009 at 3:52 PM | Permalink

    Problem has been fixed.

  7. stevemcintyre
    Posted Nov 28, 2009 at 3:54 PM | Permalink

    For those who say that blog posts aren’t reviewed: I mislocated the yellow line initially. Within 10 minutes, 4 different readers reported the error.

  8. Posted Nov 28, 2009 at 3:59 PM | Permalink

    Steve, are you aware of the current release?

    quote:
    CRU climate data already ‘over 95%’ available (28 November)

    This statement was released on November 24 at 3.30pm
    endquote

    There are two dates. Have they updated the release?

    Steve

  9. Jack
    Posted Nov 28, 2009 at 4:00 PM | Permalink

    What is the caption for this figure? It’s hard to determine if the figure is misleading when the caption that was included with it is not presented as well.

  10. Jack
    Posted Nov 28, 2009 at 4:02 PM | Permalink

    Sorry – never mind. I found it in the link you included in your post…

  11. Jean S
    Posted Nov 28, 2009 at 4:06 PM | Permalink

    Jack, although I do not understand how any caption would change the situation, here is the link to the original WMO statement:

    http://www.wmo.ch/pages/prog/wcp/wcdmp/statemnt/wmo913.pdf

  12. Jeff C.
    Posted Nov 28, 2009 at 4:08 PM | Permalink

    I had commented on this subject at the the “New data from the decline” post after Steve had questioned the smoothing methodology. Although the Briffa recon was truncated at 1960, the problems appear to start up to a decade earlier due to the smoothing methods that must have been applied to the data. Although I can’t say for sure, I think they padded the endpoints with zeroes (the anomaly reference) to allow the smoothed trace to continue to 1960. I would have thought the series mean of -0.33 deg would have been more appropriate value for padding.

    Endpoint padding with zeroes results in a convenient inflection that appears to show the decline reversing despite the trend in the raw data. The link below was an attempt to match the IPCC trace (a little different from what is shown here), but I think the same game is being played. Compare the smoothed to raw data.

  13. stevemcintyre
    Posted Nov 28, 2009 at 4:09 PM | Permalink

    I’ve added in the caption and link. Thanks.

  14. David
    Posted Nov 28, 2009 at 4:10 PM | Permalink

    Also:

    Say you’ve released “the data:”

    http://www.realclimate.org/index.php/archives/2009/11/wheres-the-data/

  15. Jack
    Posted Nov 28, 2009 at 4:11 PM | Permalink

    Jean, thank you for the link. It would make a difference if it were clear from the caption what the figure was comprised of. Disappointingly, it’s not clear from the caption so I agree it’s not a good figure. I don’t agree that the figure is intentionally “hiding” anything, but you must be clear about what data you are using and how you are using it.

  16. stevemcintyre
    Posted Nov 28, 2009 at 4:12 PM | Permalink

    Jeff, the smoothing efforts are much appreciated. It’s never easy figuring out what Team authors do. Have you taken a look at Jones diagram in the press statement of Nov 24. Even now, it seems that he’s hiding part of the decline. I’m going to do a post on trying to replicate this diagram.

  17. stevemcintyre
    Posted Nov 28, 2009 at 4:13 PM | Permalink

    Jack, have you read the emails in the two months leading up to the “trick” email?

  18. Jack
    Posted Nov 28, 2009 at 4:17 PM | Permalink

    Steve, no, I have not read the full suite of emails – I’ve only read the ones that have been frequently quoted and have read the responses by the scientists involved. I won’t comment further at this point since I’ve not had time to read through them all.

  19. Billthetechnicalwriter
    Posted Nov 28, 2009 at 4:22 PM | Permalink

    Let me understand: The tree-ring data post 1960 does NOT represent Temperature, however, the tree-ring data up to 1960 supposedly DOES represent Temperature. Maybe the answer to the Divergence “problem” is obviously “hidden in plain sight” – that is: NONE OF THE TREE RING DATA REPRESENT TEMPERATURE DATA at all. Sorry, paleodendroclimatologists – it is time for you to select an honest branch of science in which to end your careers (snip if desired).

  20. AC
    Posted Nov 28, 2009 at 4:28 PM | Permalink

    (this is a repost from an earlier thread – not sure if that’s ok but I’m interested in responses and it seemed relevant)

    Just an interesting thing I noticed. At the UEA website, they’ve posted some responses. At the bottom of this page UEA response
    are two graphs, labeled as follows:

    The WMO1999 figure (top) with climate reconstructions and instrumental temperatures merged, and a version (bottom) with the climate reconstructions (coloured) and instrumental temperatures (annual & summer in black) shown separately.

    If you overlay the two graphs, it’s clear that the top graph, with the items “merged” has a higher end point than the instrumental line on the second graph. Any ideas as to how merging instrumental data points with lower climate reconstruction values results in datapoints that are higher than the instrumental values? See the overlay here:

    1009061939

    The faint blue line in the upper right corner is the “merged value”. The two original graphs are available at the UEA site for anyone to work with.

  21. Dirk
    Posted Nov 28, 2009 at 4:42 PM | Permalink

    I can’t believe it’s not universally accepted that CRU has BAD SCIENCE. Because the scientific method means presenting your data and methods such that it can be replicated. If you submit a Chem 101 lab without data and calculations, you flunk. How anyone can submit climate “findings” without data and calculations to allow others to go where they went and pull cores like they pulled and get equal results is beyond me. If you can’t replicate it, you can’t consider it real.

    When I was an engineer in industry, I qualified materials for use, often seeking cost reduction. I remember a couple times that the data did not show equivalent reliability- usually not all the samples were
    “bad”, just a few- so we couldn’t use the material until understanding why the “bad” data points were “bad”. We couldn’t just hide them.

    Thank goodness these guys are working on the weather and not food additives or building materials.

  22. Doug Badgero
    Posted Nov 28, 2009 at 4:58 PM | Permalink

    I think focusing too much on the motives of those involved misses the point and provides an opportunity for UEA and others to dismiss legitimate criticisms. The issue is ‘how well do pre c.1850 proxies predict what temperature was at the time?’ We know the following:

    1. The Briffa tree ring proxies don’t work after 1960.
    2. Strip bark trees are poor temperature proxies and were used extensively in this reconstruction.
    3. Unsurprisingly, many of the data sets used to create the graph fail R-squared tests: “most dismally” – Steve M.

    The UEA statements have mostly attempted to claim that data has been available to the “skeptics” all along. Don’t let them change the debate too far away from the validity of the scientific methods they used in the first place. Let the politicians and pundits debate their motives – they don’t matter that much to the real scientific debate.

  23. Bill Jamison
    Posted Nov 28, 2009 at 5:09 PM | Permalink

    Did anyone else see the report in the Daily Telegraph that claims CRU has decided to make their data publicly available?

    Looks like a victory for Steve and everyone else that has been trying to get the data from CRU!

    http://www.telegraph.co.uk/earth/copenhagen-climate-change-confe/6678469/Climategate-University-of-East-Anglia-U-turn-in-climate-change-row.html

  24. Pops
    Posted Nov 28, 2009 at 5:21 PM | Permalink

    Is there anything left in the CRU cooking pot?

    …This weekend it emerged that the unit has thrown away much of the data. Tucked away on its website is this statement: “Data storage availability in the 1980s meant that we were not able to keep the multiple sources for some sites … We, therefore, do not hold the original raw data but only the value-added (ie, quality controlled and homogenised) data.”…

    http://www.timesonline.co.uk/tol/news/environment/article6936289.ece

  25. David
    Posted Nov 28, 2009 at 5:24 PM | Permalink

    It looks like a comment I posted earlier in this thread has been skipped over and is still awaiting moderation.

  26. Ian
    Posted Nov 28, 2009 at 5:56 PM | Permalink

    The problems with paleo-climatology were actually well recognized (though not publicised) by members of the Team. Ed Cook proposed to Briffa that they jointly write a paper entitled:

    “Northern Hemisphere Temperatures Over The Past Millennium: Where Are
    The Greatest Uncertainties?”

    See: 1062592331.txt

    The email is very interesting – it shows the fear that exists about potentially stepping outside of the Team circle (which is one of the saddest aspects revealed by the emails):

    “- I am afraid the Mike and Phil are too
    personally invested in things now (i.e. the 2003 GRL paper that is
    probably the worst paper Phil has ever been involved in – Bradley
    hates it as well), but I am willing to offer to include them if they
    can contribute without just defending their past work – this is the
    key to having anyone involved. Be honest.

    Here are my ideas for the paper in a nutshell (please bear with me):

    … [He lists 5 items, then comes to the anticipated conclusions:]

    6) Point out implications concerning the next IPCC assessment and EBM
    forcing experiments that are basically designed to fit the lower
    frequencies – if the greatest uncertainties are in the >100 year
    band, then that is where the greatest uncertainties will be in the
    forcing experiments

    7) Publish, retire, and don’t leave a forwarding address

    Without trying to prejudice this work, but also because of what I
    almost think I know to be the case, the results of this study will
    show that we can probably say a fair bit about 100 year variability was like with any certainty (i.e. we know
    with certainty that we know f[***]-all).” [Edited: it is a family blog, after all!]

    So, they fully recognize the problems – the very ones that Mr. McIntyre has tirelessly raised – but are too intimidated to point them out.

    The email notes in conclusion:

    “If you don’t want to do it, just say so and I will drop the whole
    idea like a hot potato. I honestly don’t want to do it without your
    participation. [...]”

    Such is the state of the science in paleo-climatology. It’s truly tragic.

  27. Pistolus
    Posted Nov 28, 2009 at 6:03 PM | Permalink

    Mann looks like he is ready to throw Jones under the bus, from his missive to reporters and others defending himself (and only himself) Mann writes the following (reproduced by Joe Romm):

    “3. “Can you delete any emails you may have had with Keith re AR4? Keith will do likewise. He’s not in at the moment -minor family crisis. Can you also email Gene and get him to do the same? I don’t have his new email address. We will be getting Caspar to do likewise.” (from Phil Jones)

    This was simply an email that was sent to me, and can in no way be taken to indicate approval of, let alone compliance with, the request. I did not delete any such email correspondences.”

    http://climateprogress.org/2009/11/28/climategate-michael-mann-hockey-stick-copenhagen-diagnosis/#more-14670

    He’s got some other lengthy explanations there also.

  28. Hans Palmstierna
    Posted Nov 28, 2009 at 6:23 PM | Permalink

    Let me see if I understood this correctly :

    a ) Briffa creates his dataset, and realizes that data after 1960 diverge from temperatures
    b ) Briffa decides that there is a “non-temperature-related” signal in the data series causing this divergence
    c ) Briffa explains in a paper that the “decline” is a problem, and thus data after 1960 should not be used to represent temperatures
    d ) Briffa does NOT explain how the remaining data is still valid, and can be assumed to be a proxy for temperature
    e ) “A good way to deal with the problem” in Briffas series is to remove the decline and add to it data that moves completely opposite.

    Is this some scientific twilight zone, or what??

    Steve
    : There seems to have been pressure on Briffa. More tomorrow.

  29. Oldjim
    Posted Nov 28, 2009 at 6:28 PM | Permalink

    Jackpot!!

    http://www.telegraph.co.uk/earth/copenhagen-climate-change-confe/6678469/Climategate-University-of-East-Anglia-U-turn-in-climate-change-row.html

    Leading British scientists at the University of East Anglia, who were accused of manipulating climate change data – dubbed Climategate – have agreed to publish their figures in full.

  30. HR
    Posted Nov 28, 2009 at 6:47 PM | Permalink

    Surely the data hasn’t been “not shown” or even “hidden” but completely replaced with something different.

    What is the point of doing this? I don’t know how any scientist outside the ‘gang’ can justify this.

  31. Robert Wood
    Posted Nov 28, 2009 at 7:23 PM | Permalink

    http://www.telegraph.co.uk/earth/copenhagen-climate-change-confe/6678469/Climategate-University-of-East-Anglia-U-turn-in-climate-change-row.html

    So, UEA and The Had Crew have decided to publish that which the dog ate?

    On another note, I see some MSM outlets recognizing Climategate, but do not understand or report the significance of all the other documents, software and data files.

  32. Posted Nov 28, 2009 at 7:27 PM | Permalink

    Ian, you didn’t quite get the pertinent part of Ed Cook’s quote right. I have it here: http://whatcatastrophe.com/drupal/node/47

  33. docmartyn
    Posted Nov 28, 2009 at 7:32 PM | Permalink

    Steve, have you any idea what the correlation between the tree ring data used and the independently measured temperature, THAT WAS USED IN THE CALIBRATION PERIOD.

    This is the bit that interests me. What is the R2 for tree rings vs temp, in the bit history they claim it was correlated. They are claiming errors of +/- in the derived data. This would mean that the correlation needs to be very good indeed <0.5/20.

  34. Jeff C.
    Posted Nov 28, 2009 at 7:33 PM | Permalink

    Re: Steve McIntyre – take a look at the second figure in the 11/24 statement that is immediately below the figure you posted above. Although the Briffa recon is truncated in 1960, they don’t use the zero pad endpoint smoothing trick. The plot uses either the post 1960 data for smoothing or possibly the series mean for points after 1960 when smoothing.

    In the second figure the green trace ends in 1960 at the same level as the plateau seen around 1900 (around -0.22 deg on the plot). In the spliced version you show above and in the 2001 IPCC plot, the inflection that happens in between 1955 to 1960 is a good 0.1 deg above the 1900 plateau (around -0.1 deg on the plot).

    They used the zero-pad smoothing trick to hide part of the decline that happens before 1960 and also to get the trace heading the right direction for the splice. This was a second trick in addition to the post 1960 truncation.

    I’m surprised they posted the second figure as it seems to make the smoothing traick obvious.

  35. Jonathan Fischoff
    Posted Nov 28, 2009 at 7:36 PM | Permalink

    Explanations like the ones they are pitching only work for the ignorant. Unfortunately for the cause, they are motivating intelligent informed people to be skeptical.

  36. Jeff C.
    Posted Nov 28, 2009 at 7:51 PM | Permalink

    Here is a link that compares the endpoints of the Briffa recons from two different sources. In both cases, the traces end in 1960. The top plot is from the IPCC 2001 report, the bottom one is from the UEA press release of 11/24/2009 while in full damage control mode.

    Steve: in fact, the Briffa decline is “worse than we thought”. More tomorrow.

  37. marie elks
    Posted Nov 28, 2009 at 8:12 PM | Permalink

    Untitled Document

    Mr Steve–
    Have you looked at the file SOAP-D15-berlin-d15-jj.doc in the documents folder
    of the .zip file? Does this sound even reasonably sane? To an average person
    it sounds quite insane.

    At the same time, a series of millennial climate simulations was conducted
    using estimates of solar irradiance (S), greenhouse gases (G), and volcanic
    aerosols (V) as external historical forcing. Ideally, the internal (=natural)
    fluctuations (the “noise”) as well as the forcing response (the
    “signal”) is of the same scale (and shape) in all models and
    equals reality. And with a sufficiently large signal-to-noise ratio
    (SNR) for a given forcing it should be possible to distinguish the signal
    from the noise and determine the sensitivity of the climate system to that
    forcing; moreover, proxies and models would show the same signal. In the
    real world, however, both noise and signal tend to depend on the specific
    model, and since proxy reconstructions are imperfect as well the real signal
    is deeply obscured.

    [image]

    Fig. 2 shows two millennial simulations using all three forcings (S, G,
    and V), one from the Hadley centre model HadCM3 (“all” simulation)
    and the other from the DKRZ model ECHO-G (“Erik” simulation).
    Obviously both models deviate considerably from each other and from
    the instrumental record (Jones et al., 1999), with a cold bias of as much
    as 1K (ECHO-G) and 1.5K (HadCM3). This is compensated for by a tropical
    and SH warm bias (not shown) which gives a much better fit for the global
    average.
    At least some of the Northern Hemisphere difference between
    the two simulations may be because the HadCM3 all simulation has aerosol
    forcing, whereas the Erik simulation does not (Note that GCM parameters
    are tuned using that average, among others, as a target.) It is unlikely
    that from those raw time series a forcing signal can be detected.

    [image]
    However, under rather mild assumptions it can be assumed that the
    response of the system will not change drastically if transfered from the
    too cold to the normal state. Therefore, it is justified to replace both
    simulations with a corresponding best estimate to the CRU data, just like
    the proxies in Fig. 1
    .

  38. Jeff C.
    Posted Nov 28, 2009 at 8:29 PM | Permalink

    Steve –looking forward to tomorrow’s post!

    I plotted this up so I’ll throw it out there for review. This plot compares Briffa’s raw data smoothed data using zeroes for end padding vs. using the post-1960 data for end padding:

    The zero end padding trace looks similar to the IPCC plot shown here:

    http://camirror.wordpress.com/2009/11/26/new-the-deleted-data/

    The post-1960 data end point padding trace looks like the second plot from the UEA press release of 11/24/2009 here:

    http://www.uea.ac.uk/mac/comm/media/press/2009/nov/homepagenews/CRUupdate

    Why would they be different? I might guess the 11/24 plot was rushed out in a panic and they forgot to check it against stuff released previously.

  39. curious
    Posted Nov 28, 2009 at 8:30 PM | Permalink

    Oldjim – not so sure about the “Jackpot!!”. IMO there are two lemons on the line at the moment and the third is maybe within nudging distance. Next thing is: What is inside the machine to payout?…

  40. Posted Nov 28, 2009 at 8:41 PM | Permalink

    RealClimate, which is the voice of Hadley-CRU-IPCC, has a list of publicly available data – they are moving on, and supposedly there is lots of other evidence of imminent doom unless we repent of our sins against Gaea.

    My understanding is that most of this data is not raw data, and such of it that is raw data, fails to support the graphs of doom.

    I believe this, since real climate does not produce an example of a graph of doom that can be derived from raw data and public code, apart from ten trees that look suspiciously like cherry picked data and fail to agree with a larger tree sample. Is my assumption valid? Do, they in fact have anything to move on to?

  41. docmartyn
    Posted Nov 28, 2009 at 8:56 PM | Permalink

    Malcolm Hughes- I am confident that, before AD1850, (stripbark’s) do contain a record of decadal-scale growth season temperature variability. I am equally confident that, after that date, they are recording something else.

    Alleged CRU Email – 969618170.txt

    Quoting tom crowley :

    Dear Malcolm and Keith, as I discuss in my Ambio paper the “anomalous” late 19th century warming also occurs in a LaMarche tree ring record from central Colorado, the
    Urals record of Briffa, and the east China phenological temperature record of Zhu.

    Alpine glaciers also started to retreat in many regions around 1850,
    with 1/3 to 1/2 of their full retreat occurring before the warming that
    commenced about 1920.

    The Overpeck et al Arctic synthesis also discusses warming before 1920 -that record matches very closely the Mann et al reconstruction in other details back to 1600.
    Unpublished work by us on coral trends also suggests slight warming between about 1850-1920.

    So, are you sure that some CO2 fertilization is responsible for this?
    May we not actually be seeing a warming?

    Tom
    – – – – – – – – – — – — – – – – –
    Thomas J. Crowley From: mhughes@xxxxxxxxx.xxx
    To: tom crowley
    Subject: Re: old stuff
    Date: Fri, 22 Sep 2000 06:22:50 -0700
    Cc:

    Dear Tom,
    The difference between the Campito Mountain record and, for example, the one
    from the Polar Urals that you mention, is that there is no meaningful
    correlation between the Campito record and local temperature, whereas there is a
    strong correlation in the Polar Urals case. I give references to the work
    reporting this phenomenon at the end of this message, but I’m afraid I’m missing
    the references to the technical comments that are being responded to in the last
    two. If you examine my Fig 1 closely you will see that the Campito record and
    Keith’s reconstruction from wood density are extraordinarily similar until 1850.
    After that they differ not only in the lack of long-term trend in Keith’s
    record, but in every other respect – the decadal-scale correlation breaks down.
    I tried to imply in my e-mail, but will now say it directly, that although a
    direct carbon dioxide effect is still the best candidate to explain this effect,
    it is far from proven. In any case, the relevant point is that there is no
    meaningful correlation with local temperature. Not all high-elevation tree-ring
    records from the West that might reflect temperature show this upward trend. It
    is only clear in the driest parts (western) of the region (the Great Basin),
    above about 3150 meters elevation, in trees old enough (>~800 years) to have
    lost most of their bark – ‘stripbark’ trees. As luck would have it, these are
    precisely the trees that give the chance to build temperature records for most
    of the Holocene. I am confident that, before AD1850, they do contain a record of
    decadal-scale growth season temperature variability. I am equally confident
    that, after that date, they are recording something else.
    I’m split between Harvard Forest and UMASS these days, and my copy of your paper
    is not with me today. I’d be interested to know what the name of the site for
    the LaMarche central Colorado record was.
    Cheers, Malcolm

  42. Ian
    Posted Nov 28, 2009 at 9:55 PM | Permalink

    Jeff:

    My bad: the copy & paste didn’t quite work as well as it should have…sigh. Still getting used to putting stuff together in these little comment boxes…

    “Without trying to prejudice this work, but also because of what I
    almost think I know to be the case, the results of this study will
    show that we can probably say a fair bit about 100 year variability was like with any certainty (i.e. we know
    with certainty that we know f***-all).”

  43. Posted Nov 28, 2009 at 10:30 PM | Permalink

    A few hours after promising to release the data, according to the UK Sunday Times, the university was admitting that all the raw data from the 80s and before had been dumped when they moved locations:

    ““SCIENTISTS at the University of East Anglia (UEA) have admitted throwing away much of the raw temperature data on which their predictions of global warming are based.

    It means that other academics are not able to check basic calculations said to show a long-term rise in temperature over the past 150 years.

    The UEA’s Climatic Research Unit (CRU) was forced to reveal the loss following requests for the data under Freedom of Information legislation. “

    The data were gathered from weather stations around the world and then adjusted to take account of variables in the way they were collected. The revised figures were kept, but the originals — stored on paper and magnetic tape — were dumped to save space when the CRU moved to a new building.””

    It’s okay though, because they have the ‘value added, homogenised and quality controlled data.’ And they say so, thus it must be true, yes?

  44. AntonyIndia
    Posted Nov 28, 2009 at 10:31 PM | Permalink

    The Finnish dendroes do NOT see a hockey stick in their wood; their line does not freak out after 1960. Their line is 7641 years long by the way.
    see at http://lustiag.pp.fi/ (all in English, no worries).

    Old news for Steve Mc. but surely interesting for “the bigger audience” now.

  45. Ian
    Posted Nov 28, 2009 at 10:34 PM | Permalink

    Ok, that didn’t work either…it looks right when I put it in, but it loses the signs…

    Without trying to prejudice this work, but also because of what I
    almost think I know to be the case, the results of this study will
    show that we can probably say a fair bit about [less than] 100 year
    extra-tropical NH temperature variability (at least as far as we
    believe the proxy estimates), but honestly know fuck-all about what
    the [greater than] 100 year variability was like with any certainty (i.e. we know
    with certainty that we know fuck-all).

  46. John A
    Posted Nov 28, 2009 at 10:45 PM | Permalink

    I’ve just had a look at “Doltoid” and it looks like Tim Lambert is trying to bail out the incoming tide with a beach bucket.

  47. Andrew
    Posted Nov 28, 2009 at 11:05 PM | Permalink

    All the excitement regarding the forthcoming data will (I predict) end up as nothing more than a (hopeful on the CRU part) Red Herring. Unless they are will ing to give up all the Raw data, method, etc. it amounts to a hill of nothing. Again it is most important to ask now, through what ever means possible, for this. CRU will likely give the data that makes up their graphs and thus point to the releasing of the data as as means to shut up the skeptics. “There. We have given you the data….now go away…”. I’m sure Steve M will be in contact with CRU to discuss what is coming….well I hope so. Steve have they been in contact? You with them?

  48. David
    Posted Nov 28, 2009 at 11:14 PM | Permalink

    A very interesting (and super long) reconstruction of e-mail context by John Pittman:

    http://noconsensus.wordpress.com/2009/11/28/context-2/

    Plus, Jeff Id has edited excerpts from something you wrote into the past into a strongly worded post called “Sesame Street Science:”

    http://noconsensus.wordpress.com/2009/11/27/sesame-street-science/

  49. effinayright
    Posted Nov 28, 2009 at 11:39 PM | Permalink

    Can someone help a poor layman out here? When I was a kid in college an elec. engineering prof used to have us do quick problems in class. Once , when a student gave “the answer” as “6.346 Ohms”, the prof said “Say, that’s some slide rule you got there”. The problem, of course, was the precision that could be derived from a slide rule.

    So I’m asking how all this world-wide data supposedly “measured” over a thousand of years or more can conveniently be precise to the tenth of a degree, and “smoothed” at that.

    I’m told that areas of the earth for which there were no temperatures collected were assigned values . Well, if so, how could anyone assign a value, to a tenth of a degree? How can tree-ring proxies, given the variations in rings due to soil characteristics, water, humidity, cloudy-vs-sunny weather, etc. be measured with such precision?

    Can someone point me (and others ) to a site or article explaining how such precision is achieved, given the imprecision (or downright inaccuracy) of early temp measurements, directly or through proxies?

    Or is that just one of the issues McIntyre et al have been raising?

  50. OzzieAardvark
    Posted Nov 29, 2009 at 12:20 AM | Permalink

    Given the turn this thread has taken, it seems appropriate to post up what I posted at WUWT a few minutes ago:

    Hoping for anything from Penn State is simply wishful thinking. What would be more interesting is whether the Climategate political fallout will reach Dr. Mann. A useful thing in this regard would be a subpoena that requests what e-mails he sent following his response to Phil Jones’ request that he delete e-mails related to AR4. The two e-mails follow:
    ————————-
    Phil Jones wrote:
    Mike,
    Can you delete any emails you may have had with Keith re AR4? Keith will do likewise. He’s not in at the moment – minor family crisis.
    Can you also email Gene and get him to do the same? I don’t have his new email address.
    We will be getting Caspar to do likewise.
    I see that CA claim they discovered the 1945 problem in the Nature paper!!
    Cheers
    Phil
    ————————-
    Michael Mann replied:
    Hi Phil,
    laughable that CA would claim to have discovered the problem. They would have run off to the Wall Street Journal for an exclusive were that to have been true.
    I’ll contact Gene about this ASAP. His new email is: generwahl@xxxxxxxxx.xxx
    talk to you later,
    mike
    ————————-
    So did Dr. Mann follow-up on his commitment to contact “Gene”?
    An interesting question that I hope will be answered. I don’t expect that Dr. Mann or Penn State will come forward. That said, the individual and the institution may want to consider the implications of not doing so and the consequences of same being made public via subpoena. Congress can do that and underestimating Senator Inhoff may be… Choosing Poorly :-)

    OA

  51. DWhite
    Posted Nov 29, 2009 at 12:59 AM | Permalink

    effinayright said:
    [i]How can tree-ring proxies, given the variations in rings due to soil characteristics, water, humidity, cloudy-vs-sunny weather, etc. be measured with such precision?[/i]

    Hey, it’s Climate Science. If you like the one about 0.1 degree precision, have you heard the one about “teleconnection”? That’s the one where your tree, say in the western US, doesn’t respond to the local climate surrounding that tree, but does respond to the globally averaged temperature. No, I’m not making that up.

    I think there’s really only one way an organization can get away with such tripe, but that gets us into politics, which are off limits here. The whole thing is really quite breathtaking.

  52. John Fish
    Posted Nov 29, 2009 at 1:19 AM | Permalink

    A couple of stories in tomorrows Sunday Times (London).

    http://entertainment.timesonline.co.uk/tol/arts_and_entertainment/books/article6936404.ece

    http://www.timesonline.co.uk/tol/news/environment/article6936328.ece

    I made a comment about Phil Jones’ threat in 2005 to “delete the (CRU station data) file” if the MM’s ever heard of the UK FOI act, and it hasn’t appeared.

    Still nothing on the BBC which is a disgrace.

  53. OzzieAardvark
    Posted Nov 29, 2009 at 1:23 AM | Permalink

    snip – politics

  54. Will
    Posted Nov 29, 2009 at 2:25 AM | Permalink

    THIS IS A CALL TO ACTION
    We have been given a powerful tool in the form of GlimateGate.
    It now has a name and has the potential to get a life of its own.
    So if the mains stream media is not going to report on this then let us use the social Facebook and emails to spread the news.
    Send the following two YouTube videos to two people that you know and ask them to send it onto at least 2 others.

    And

    If you have a Facebook page post the two links.

    Will

  55. OzzieAardvark
    Posted Nov 29, 2009 at 2:59 AM | Permalink

    Uh… Interstate 44. Sorry :-)

  56. Johan Branders
    Posted Nov 29, 2009 at 3:00 AM | Permalink

    Jeff C: You wrote in the “New!! Data from the Decline” thread:
    “Not only did they “hide the decline” by truncating post 1960 values, they used very questionable smoothing methods to suggest that the drop that started around 1940 was reversing itself by 1960. This is despite the fact the raw data to 1960 shows a continuous downward decline.”

    This figure shows this in detail:

    Mark the endpoints of the other recons (red and blue) are inflected as well.

  57. vg
    Posted Nov 29, 2009 at 3:19 AM | Permalink

    might this help?
    #

    This story from the London Times forthrightly states that the raw data that Phil Jones and his team depends upon has been lost.

    http://www.timesonline.co.uk/tol/news/environment/article6936328.ece

    Can you clarify whether this is true or not.
    Thanks

    [Response: No. The original data is curated at the met services where it originated. - gavin]

    Comment by Bernie — 28 November 2009 @ 11:34 PM

  58. Jonathan Fischoff
    Posted Nov 29, 2009 at 3:51 AM | Permalink

    re: vg
    Do you know if one can download the raw data from the met?

  59. galileonardo
    Posted Nov 29, 2009 at 4:16 AM | Permalink

    I just posted this over on WUWT, but on the subject of the lost raw data, have no fear. Jones wrote the following in his 1255298593.txt email to Rick Piltz:

    “The original raw data are not lost either. I could reconstruct what we had from some DoE reports we published in the mid-1980s. I would start with the GHCN data. I know that the effort would be a complete wate of time though. I may get around to it some time. As you’ve said, the documentation of what we’ve done is all in the literature.”

    Sounds like he may be forced to waste some time after all.

  60. Posted Nov 29, 2009 at 5:19 AM | Permalink

    Jeff C thanks for inspiration to look closely. Here follows what I’ve garnered – open to corrections from readers of course!:

    Hat trick – more tricks

    UEA and CRU have not, it seems, stopped their tricks. Here, at the bottom of their recent release, are two graphs. The first graph was prepared for the WMO and it shows three separate proxy temperature reconstructions (from Jones et al, Mann et al, and Briffa et al) smoothly spliced to the proper thermometer record. The second looks like it has been prepared in a hurry… separating the coloured proxy curves from the black temperature curves.

    Superimpose the second graph on the first to glimpse the differences

    Home in closer – the first is to “hide the decline” but – oops – it’s clearly there in all recons when the thermometer records are separated out.

    Home in still closer to see six tricks.
    (1-3) Each proxy record starts to show a decline from 1960 on, the red (Jones), the blue (Mann), the green (Briffa).
    (4) The splices used in the original graphs SOAR up to year 2000 – to outdo both the proxies and the thermometer record. WTF??
    (5) The original is said to be anomalies from 1961-1990 baseline. But when we see the 1961 and 1990 lines, the baseline looks way off. WTF??
    (6) There are TWO black temperature lines. The notes say instrumental temperatures (annual & summer in black) shown separately but summer temperatures should be consistently higher than annual, not criss-crossing as here. WTF??

  61. Shona
    Posted Nov 29, 2009 at 6:00 AM | Permalink

    According to Courtillot, tree, not rings, but density are extremely good proxies for temp when temp is changing rapidly, however when temp plateaus, then trees adapt and they no longer show temp well. (As someone who has grown plants for a long time this “sounds” right, in normal conditions they aren’t too sensitive to temp, but react strongly when temp changing significantly.)

    I’ll look for the place in his Nantes lecture where he shows the graphs.

    Steve I seem to be caught in the spam trap

  62. bender
    Posted Nov 29, 2009 at 6:10 AM | Permalink

    Pressure on Briffa? From the likes of Jones & Mann? Never!

  63. Posted Nov 29, 2009 at 6:14 AM | Permalink

    Please delete my OTT comments at the end of some of my lines. Apologies.

  64. vg
    Posted Nov 29, 2009 at 6:22 AM | Permalink

    Jonathan: I would not have a clue. I just saw this a RC and thought it might help. maybe you could write to Gavin and he would direct you.. Though I think it might be a furfy. I hope it is possible BTW.

  65. billcat
    Posted Nov 29, 2009 at 6:48 AM | Permalink

    Somewhat OT, but Rasmus Benestad is at it again in the Norwegian research online: forskning.no

    http://www.forskning.no/artikler/2009/november/235924

    I ran the thing through google translate and tidied up afterwards. Somewhat, his original Norwegian prose itself look like it is the result of a google translate effort. :-s

    A lesson from the CRU hack?

    One should certainly be carful of what you write in e-mail to colleagues. For perhaps we really are living in 1984-the world (ie 1984 from George Orwell) in the workplace, when confidential e-mails so easily can be posted completely open?

    Even if they are supposedly be job related, the human factor also matter when you communicate with each other. Or perhaps one should always write in a stiff and formal style to all his colleagues? But who does that today?

    Not climate scientists anyway -as we have just learned, and it is quite clear that someone attempts to create a sensation out of it.

    Because some of the things that came out was certainly ugly and embarrassing. But the important point is that it was only an internal dialogue between colleagues who did not involve any outrageous actions. This is of course possible to check. The IPCC report contains the answer – which articles are missing which shouldn’t be included? [Comment: Yes, this is a direct translation] If so, should we not have heard of this before now?

    Just talk

    The analogy is two friends talking about a bank robbery, and then they are overheard and there is a big commotion. But there has not been a bank robbery, and while they only talk about it, they have not done anything illegal (even if such talk is probably not wise). Similarly, the e-mails contained a lot of talk, but no unethical action. The reason is that there are many scientific journals (I haven’t got the overview) and many scientists at a high level are not the best of friends. There are more analysis of the global mean temperature, in addition to sea-level temperatures, from other sources.

    But I also certainly had a strange feeling when I had finished reading the book ‘The Republican War on Science’ (Chris Mooney), right after the hack at the Climate Research Unit (CRU, University of East Anglia, England). In the book, Mooney discusses the climate debate along with a number of other issues such as environment contaminants, endangered species, diet-sugar [sic], intelligent design, stem cells, abortion, tobacco, and sex education. In all these cases, he finds a recurrent pattern of contempt towards knowledge and research.

    The theme of the book suddenly came much closer to home- because it feels so much more real after I myself was spun into the drama surrounding the hack at CRU. One of my emails was also published, as I had to explain to a couple of journalists. Moreover, RealClimate.org had been tried hacked in connection with this raid.

    The most obvious similarity between the CRU-break and The Republican War on Science touches legislation such as Freedom of Information Act ‘, the Shellby amendment, and the Data Quality Act, which among other things, provides industry and interest groups free access to all public data. It is a good principle, but in the wake of these there has also have been some unfortunate experiences with unreasonable arguments based on these data. Confusion is created when matter is technical, difficult, complicated and hard to get an overview of.

    Out of proportions

    We have seen ugly concrete examples that this has happened: namely, “Hockey Stick graph” of Michael Mann. This has also probably had an impact on some climate researchers’ behavior, and created a resistance among some to give out data (another aspect of the matter is that more data are commercial, and can not be given out freely to a third party). Moreover, strong personal conflict also affected the situation, and this is obviously not good.

    What also emerges is an extremely caution expressed in the e-mails. The aftermath of this CRU-break has also to the full shown that this has been prudent. Because some climate skeptics have turned an illegal hack on its head, and tried to make the victims emerge as the villains. Climate scientists are rendered suspect and their opponents blow things far out of proportion.

    When one reads the responses to Bjørnar Kjenslis reportage of the case on forskning.no (No climate conspiracy in sight) [Comment: A whitewash, nothing-to-see-here article at forskning.no, in which the journalist was skewered in the comments], it is almost frightening how unreasonable reactions on this. These show a blind witch hunt of climate scientists.

    Simply associating the break with Watergate scandal, shows how the case is turned upside down. Because it is not illegal to send e-mails to colleagues, and it was a burglary at the CRU which was illegal. Nixon however did something illegal, while the Watergate investigation in itself was not.

    Calling the situation the ‘Watergate of Climatology’ – or ‘Climate Gate’ – is also a trick that could well have been taken directly from ‘The Republican War on Science’. That there is a conspiracy when climate scientists communicate with each other is absurd – even if some scientists send some naive, unethical, or poorly thought out emails.

    Release of data

    It can nevertheless not be justified to not release the data. It is also important to build up knowledge in the poulance so that they understand what the data means and their limitations. This should be a job for us climate scientists, but the problem is that there are not very many climate scientists, and we don’t have such large resources.

    I do not think we have a good enough capacity to explain or guide if there is a high demand. Often there are people who misunderstand the data and we must correct the abuse, especially when used in a political context by parties that have the public relations expertise that only wealthy organizations can buy. The fact that you could buy T-shirts printed to ‘Hide the Decline’ – ie a reference to the hack – less than 3 days after it became known, shows how well-organized adversary climate scientists are meeting. And the short film ‘Hide the Decline’ is posted on YouTube less than 7 days after the hack was known.

    Chris Mooney writes that when the Office of Technology Assessment (OTA) was closed, the stage was set for the so-called “science courts” in the U.S. Congress. It became a play to the gallery event that gave the wrong impression of academic expertise and where the jury had no basis on which to consider the substance. But this is also apparently come to Norway, where some Norwegian political parties seem to have copied the American model.

    Nina Kristiansen[Editor at forskning.no] writes in the commentary ”stolen insight into climate research” that climate research should open all doors, also in the room where uncertainty and disagreement prevails. And I share her views on this, and she also has many points. But I think that there are also several other issues that should be highlighted. Not least, issues surrounding climate skeptics. Because, as Chris Mooney explains, there is a lot of shady politics going on in the framework and guidelines around the science. We also need much greater transparency in all the parties and the landscape around this, especially around the media, PR industry and policies that are affected.

    Media does not do its job

    The media is important because they have much power and affect so many people. Here the resources should be used properly, and therefore transparency is essential. Some examples are why is it is that debate programs on TV about climate often do not take in the scientists who actually researching the topic, but are good at grabbing in profiled skeptics. Another example, we took up in a letter to the editor to Morgenbladet “Forskning? No!”, and we have argued the case for more openness about editorial decisions in among others forskning.no.

    To me it seems that the media has not done its job. Some of us have argued for this in the Tech Weekly ( ‘Where is the critical and burrowing journalism?’, TU 4109). There are few journalists who really have an overview of people and backgrounds, and Norwegian journalists is a very rare sight on major scientific conferences. They rather prioritize sporting events.

    But I hope that forskning.no shows up on the next European Meteorological Societys annual meeting in Zurich in September 2010. There is a media & communication session there which should attract.

  66. Shona
    Posted Nov 29, 2009 at 6:51 AM | Permalink

    Terribly sorry, but I have been posting Courtillot here:

    http://camirror.wordpress.com/about/

    by mistake.

    Steve:
    I do not endorse every “skeptic” cause at this blog. My focus is on verification. My own time and energy is limited and therefore I focus on studies relied upon by IPCC. If an IPCC author or equivalent wants to post critical analysis of Courtillot here as a thread, that’s fine. However, I don’t have time to work through Courtillot; I stand behind the work that I do and don’t want the blog to be perceived as endorsing articles simply because they are “skeptical”.

  67. Mina
    Posted Nov 29, 2009 at 6:54 AM | Permalink

    Steve
    I think you will find the problem is your lacking grasp of statistics. Norwegian climate researcher Rasmus Benestad explains on ‘forskning.no'(http://www.forskning.no/artikler/2009/november/235924), a website for science news, that you do not know what you are talking about. This is Benestad’s reply to a comment about the hockey stick debate (translation by me):

    “I can explain why I am not convinced by Ross McKitrick or Steve McIntyre. Three episodes stand out:

    1. McIntyre has used a lot of energy to argue that the curve’s “hockey stick” shape is suspect and that it is a result of principal component analysis (PCA; which is a piece of linear algebra often used in data analysis). This question is way off because PCA does not change the information of the data, but reorganizes them in a way that makes them easier to handle. It would on the other hand be reasonable to ask how much of the variance the different modes explain, and how much weight they are given in the following regression analysis. So McIntyre’s statement indicate that he really didn’t know what he was talking about (another thing is that Mann did the analysis again without involving PCA and got equal results).

    2. McKitric [sic]& Michaels argued in an article in 2004 that most of the global warming could be explained by economic activity (http://www.realclimate.org/index.php/archives/2004/12/are-temperature-trends-affected-by-economic-activity/). But they had made some fundamental errors which made their conclusions invalid – they had assumed that all measures and economic data were totally independent of each other. It is one thing that the same economic factors are valid for all measures from the same countries, but you must also account for the correlation in space.

    3. McIntyre referred to a statistical test I have developed (published in 4 peer reviewed magazines, freely accessible as open source on http://cran.r-project.org – iid.test). His comment was complete nonsense [he uses a derogatory Norwegian expression that has no English counterpart, and is used when someone gives a nonsensical answer that has nothing to do with the question].

    I believe that neither McKitric[sic] or McIntyre (or Michaels) have a very good grasp of statistics – but that doesn’t stop them from being very loud. Unfortunately my impression is that much too few researches have a very good grasp of statistics.”

    So there.

    Steve: As to 1) our criticisms of Mann’s PCA were acknowledged by both Wegman and North reports. Contrary to Benestad’s false claim, we identified the effect of the PCA modes and showed that MAnn’s hockey stick derived from strip bark bristlecones, proxies known to be questionable as “temperature” proxies and which the NAS panel said should be “avoided”. There are other ways of using the flawed bristlecones besides Mann’s flawed PCA and these have been discussed and criticized on many occasions at CA. 2) I had no involvement with the Michaels paper and do not understand why any merit or lack of merit of that paper should reflect on me. 3) Could you refer me to the comment being criticized? For example, I agree with Rasmus on the following prediction made at realclimate, but do not regard it as a particularly searching observation:

    I can easily predict that the summar will be warmer than current (winter) conditions…

  68. anon
    Posted Nov 29, 2009 at 7:58 AM | Permalink

    “[Response: No. The original data is curated at the met services where it originated. - gavin]”

    I suspect that means the original forms are available either on paper or microfiche. It’s available, but you’ll have to reenter it.

    Steve: Does Gavin know that data sent from (say) Mozambique in 1981 is still curated at the met service there?

  69. William Porter
    Posted Nov 29, 2009 at 8:16 AM | Permalink

    snip: the C-word and e-word are strictly offlimits at this blog

  70. Mina
    Posted Nov 29, 2009 at 9:08 AM | Permalink

    Steve
    I have now posted your reply on forskning.no. I asked Benestad to explain why he still thinks the hockey graph is valid, and to clarify what you allegedly commented on in a nonsensical manner. I suspect Mr. Benestad feels safe to say whatever he wants because he writes in Norwegian, and also no Norwegian MSM has not”discovered” climategate yet so he can pretend nothing happened. Unless they remove my comment (Norwegian media is terribly biased) the public might actually get a chance to see what is really going on. To be continued…

  71. stevemcintyre
    Posted Nov 29, 2009 at 9:34 AM | Permalink

    New readers, please do not post comments from news articles on a thread that has a technical point. It clutters the thread.

  72. Posted Nov 29, 2009 at 9:39 AM | Permalink

    People keep trying to give Keith Briffa a pass on these revelations claiming his openness in these matters was proven. If he was so open, I wonder how the discussion of this obviously exaggerated result never crossed his mind. Certainly when Steve made the comment to the IPCC that the original data should be shown, someone had to discuss it with the originators themselves.

  73. Alexander Harvey
    Posted Nov 29, 2009 at 9:40 AM | Permalink

    Could someone please explain something to me, very slowly. Where am I going wrong?

    There are IPCC and WMO graphics that use a Briffa tree ring series.

    In the make up of the IPCC graphic:
    An original series existed that ran till 1994.
    If a 50 year (-25/+25) smooth was used, this would be good until 1994-25 = 1969.
    It was decided not to include the series in the grapphic post 1960.
    But a truncated version of a smooth which would be good until 1969 (50 yr) was not used.

    It was the data that was truncated at 1960 (A 50 year smooth now good until 1935).
    A new smooth was produced upto 1960 using 25 years of padding data (50 yr smooth).

    At some point someone knew what both the original data post 1960 were and the padded data were.
    At that point it would have been clear precisely how misleading the final smooth was (the difference between the original series post 1960 and the padding data, or the difference between the resultant smooths).
    If that be the case it is quite clearly misleading, and if some one person knew all the stages I think they be less than honest brokers.

    In the make up of the WMO graphic:
    I simply do not know what has gone on. All the smoooths seem to run upto 2000, so they are all passed their good till dates. All the references are to papers published pre 2000 so there is a lot of data added to the end of all the original series to create the smooths.
    In the case of Briffa this would require extending from 1994 till 2025 (31 years out of 50).
    But I think I am right in thinking that it be the truncated data that was actually used, and it was padded from 1960 till 2025 (65 years out of 50). If my thinking be right that does not make any sense at all. The year 2000 would be 15 years after the point were the smooth became 100% fictitious. The last years of the other smooths are more than 50% fictitious.
    If that be the case then hese are not smooths as I know them, these I would know as extrapolations.

    So which bits have Igot are wrong? I am not being rhetorical.

    Alex

    I am genuinely baffle, and I expect bamboozled.

    Alex

  74. anon
    Posted Nov 29, 2009 at 10:07 AM | Permalink

    Steve, I have no idea. Unlike some people, I’m still having trouble parsing Gavin. I’m just guessing that’s what he meant. “Originated” would seem to indicate the forms that were collected by the Met office from observers in their network. Perhaps one of his translators can tell us whether “it” refers to “the original data”, “the met services”, “No” or “Response?”

  75. EdeF
    Posted Nov 29, 2009 at 10:12 AM | Permalink

    I would think a simpler way to smooth this data would be to take a
    Fast Fourier Transform of the time series, convert it to the frequency
    domain, cut out the high frequency components and then transform it
    back to the time domain.

  76. stevek
    Posted Nov 29, 2009 at 10:30 AM | Permalink

    Stevem – do you have any type of relationship with Victor Niederhoffer ? I work for one of his friends in the hedge fund business.

    Steve: I’ve never met him. He was a famous squash player and a much better squash player than me. But I’ve played against players who would have played against him. I once played Michael Desaulniers (who I think worked with Niederhoffer) when Desaulniers was about 15 and I was about 26 and playing my very best singles. Desaulniers absolutely annihilated me. I could take my best shot, Desaulniers could have a cup of coffee and then return anything I could hit. He was on an entirely different level than me. I’ve done better in doubles than singles – doubles gives more reward to winners. I did have a win in doubles once against the Mateers who were very good. Tom Poor probably played against Niederhoffer and I played a couple of tournaments with Poor in the 1980s. Victor Harding might have played against Niederhoffer as well.

  77. Alexander Harvey
    Posted Nov 29, 2009 at 10:40 AM | Permalink

    EdeF,

    I really do not think that FFT will address the question of what to do about the end points. The data is not cyclic, so what do you do? You could use a version that only integrates on a domain restricted to the data period but then the sinusoids would loose their orthogonality and no unique transform would exist.

    As far as I am aware the only fair ways of smoothing require you to lose a period of at least one half of the smoothing period at each end. By “fair” I mean ones that do not introduce arbitary additional data or vary the length of the smoothing period.

    I think it is the question of what to do with the end points that is the basis of all the issues here and I do not see how using FFT would alter that.

    Alex

  78. pete
    Posted Nov 29, 2009 at 11:00 AM | Permalink

    Sorry off topic, but found an interesting e-mail on how CRU TS 3.0 was verified.
    Did a search on Harry:

    http://www.eastangliaemails.com/emails.php?eid=1009&filename=1252090220.txt

  79. mikep
    Posted Nov 29, 2009 at 11:26 AM | Permalink

    Mina,
    It’s a bit difficult to understand Benestad’s criticisms from your description, but the link to real climate goes to a post which itself sems to display misunderstanding. One criticism of the McKitrick Michaels work seems to have been whether spatial autocorrelation was properly treated. But rela cliamte seemed to get confused. The dependent variable (temperature by grid) did indeed show spatial autocorelation. But the residuals of the equation were white noise, and that is what matters. There is plentyof work on this at McKitrick’s site

    http://ross.mckitrick.googlepages.com/

    as well as some CA postings, which I do not have directly to hand.

  80. P Gosselin
    Posted Nov 29, 2009 at 11:39 AM | Permalink

    The news reports are encouraging today.
    snip – please do not go a bridge too far
    Websites, especially CA, did the heavy number crunching.
    And Great Thanks to the NEW MEDIA for getting the story out.
    1. UK Telegraph
    2. Drudge Report
    3. Hot Air
    4. American Thinker
    5. Talk Radio, yes especially Rush Limbaugh, love him or hate him
    6. FOX NEWS
    7. And all the other numerous sceptic blogs, there are so many I could never list them all.

    I’m a man of modest means, and the financial crisis has hit me too. But I’m going to leave a tip to both CA and WUWT. We truly do have something to be thankful for this Thanksgiving season. Thanks again.

  81. PaulM
    Posted Nov 29, 2009 at 12:03 PM | Permalink

    The graphs posted at UEA seem to do nothing for them except expose the “hide the decline” trick. Their rambling, desperate comments are a PR disaster.

    Very good coverage in the Sunday Times today. Two articles by their enviro correspondent, one about the admission of raw data and one about the whole email affair, and another article about how CRU need to engage with critics.
    The BBC is still in denial.

    Some are saying its good that they are saying that they will release the data. But theyve been promising that for years, right?

  82. Adam Gallon
    Posted Nov 29, 2009 at 2:52 PM | Permalink

    Let’s assume for a moment, that the graph’s a good representation of the historical facts. I wonder what current temperatures & climate would be like without the increase?
    We’d be what, half a degree below the 1961-1990 normal?
    A little snippet in The Daily Telegraph noted an excess death rate last winter in the UK of 30-odd thousand.
    How many more deaths would we be seeing?

    Interestingly, here’s today’s “most viewed” list!
    1.Climate change: this is the worst scientific scandal of our generation
    2.Climategate: University of East Anglia U-turn in climate change row
    3.Tiger Woods ‘told friend his wife had gone ghetto on him’
    4.Climategate e-mails sweep America, may scuttle Barack Obama’s Cap and Trade laws
    5.The curious incident of Tiger Wood’s crash in the night-time poses puzzling questions

  83. ThinkingScientist
    Posted Nov 30, 2009 at 9:38 AM | Permalink

    I agree with the comments by Alexander Harvey. If you are applying a centred smoother than the last valid point is the last year in which the full filter is applied. You cannot know what a centred 50 year average will look like until you have a further 25 years of data. Any attempt to do so is arbitrary and misleading.

    If you want to try and run the filter to the end you might taper the filter ie progressively shorten the filter length as you approach the end of the series. However this then means that the ends of the time series change from being a low frequency estimate to a broad band estimate. Still misleading.

    Padding the end of the time series with any particular value is arbitrary and essentially changes the stationarity assumption. Of course if you want to get a particular result then end padding with a “preferred value” is the method of choice.

    On the question of filtering in the frequency domain (via FFT) rather than in the time domain, you still have problems. If you frequency domain filter than the trauncation in the frequency (or slope, if its not too severe) will create a “ringing” effect when you back transform to the time domain. This can be particularly noticeable if you have a large transient (eg 1998 el Nino) near the end of the series.

    All methods of filtering suffer from side effects at the end of the series. You can’t know something that hasn’t happened yet, no matter how hard you wish it.

  84. Henry
    Posted Jan 6, 2010 at 7:20 AM | Permalink

    The UEA statements seem to have moved. The one mentioned in this thread is now here though the two charts at the end now seem to have got lost. One (the WMO version) is at the top of this thread.

One Trackback

  1. [...] The LA Times omits clear facts, offering instead the conventional he-said she-said political narrative. This tells the reader nothing that;’s not already widely known. To get that information, you have to go to blogs that do in-depth analysis. [...]

Follow

Get every new post delivered to your Inbox.

Join 3,329 other followers

%d bloggers like this: