Will the Real Slim Shady Please Stand Up? Re-Mix.

There’s an amusing little incident with the deleted “original” data set that was posted up for a few minutes at Mann’s website – you know, the data set that was first demonstrably referenced by a CA reader in the early morning of Sep 5. (I’ll reserve comment for now on issues relating to the timestamp of this data set and the Gavin Schmidt hyperlink to it, presently pointing to a data version that did not exist at the time that the hyperlink was supposedly created.)

Within a day, on the afternoon of Sep 5, the data set was deleted and replaced with another data set, again without notice, in a bewildering concatenation of replacements that is reminiscent of our experience with the Hansen’s GISS data almost a year to the day ago. However, both myself and others took the precaution of downloading the Sep 4 version as soon as we saw it – just in case it disappeared. Not an imprudent precaution, given its almost immediate deletion.

I’ve now had an opportunity to forage through the deleted version. The deleted data had 1357 series, from which 148 series were deleted to yield the 1209 series that now appear in the “original” data. But surely the 1357 series is “more” original than the 1209 series? What criteria were used to winnow out the 148 removed series? Inquiring minds want to know. There’s not a whisper on this topic in the paper or in the SI and, of course, all traces of the 148 series were ruthlessly scrubbed from Mann’s website.

Surprisingly, on the list of deleted series was a series entitled “Yamal 2002″. Now Yamal is very familiar to CA readers, as Briffa replaced the Polar Urals update (with a high MWP) with his Yamal version (with a HS) and this one substitution affects a number of reconstructions (discussed on many occasions:

The graphic below illustrates the difference between the Polar Urals update (black) and Yamal (red) (See above links for the impact of the Yamal substitution on the Briffa 2000 reconstruction, a spaghetti graph)


Figure 1. Comparison of Polar Urals update to the Briffa Yamal version.

So why was this series excluded from Mann et al 2008? And if it was no good for Mann et al 2008, why is it any good in the “other” studies?

Just for fun, I plotted up the Mann 2008 version of Yamal (shown in the graphic below):


Figure 2. Plot of “Yamal 2002″ in deleted Mann et al 2008

Quite obviously this data set doesn’t have a HS and doesn’t look anything like the Briffa version of Yamal. Where did it come from? This data version is the Yamal version from the original authors (available at WDCP here ). This version matches the graphic in Hantemirov 2002. (In passing, I’ve observed previously that Juckes et al and other Team articles have cited Hantemirov, when they actually used a Briffa version that is not located at WDCP.)

Where did the Briffa Yamal version come from and why is it different than Hantemirov’s? Presumably they use different standardization methods, but then one would have to see the measurement data, which Briffa thus far has steadfastly refused to disclose (for 8 years and multiple articles.)

I still don’t know why the Hantemirov version wasn’t used in Mann et al 2008. Would it “matter”? Yes and no. As a single series, it wouldn’t necessarily “matter”, but it would be nice to know why these series were excluded. Curiously, Mann used Briffa’s Tornetrask version (not Grudd’s), so it’s not like he refused to use grey Briffa versions. It’s odd that the Hantemirov version got into the mix in the first place.

Now there’s a nice little ending to this story. The “wrong” data set – the one deleted from Mann’s website – is the one that Mann sent to WDCP, an archive of record, where Mann can’t change “inconvenient” data sets without leaving a trace. So WDCP has one “original” data set and Mann’s website has another.

Will the real Slim Shady please stand up? (Re-Mix).

73 Comments

  1. Posted Sep 12, 2008 at 7:25 PM | Permalink | Reply

    Just as a matter of interest, what if a mining exploration company put up the results of its split core analyses as part of its responsibilities to the stock market, and then suddenly and without warning cut down the number of analyses posted which had the effect of inflating the expectation of mineral yield.

    Would this hypothetical exploration company be allowed to get away with such a substitution?

  2. John Baltutis
    Posted Sep 12, 2008 at 7:26 PM | Permalink | Reply

    For those so inclined, the “original” data set is at

  3. John Baltutis
    Posted Sep 12, 2008 at 7:27 PM | Permalink | Reply

    Link adding is broken. The complete set is at: ftp://ftp.ncdc.noaa.gov-1/paleo/contributions_by_author/mann2008

  4. jae
    Posted Sep 12, 2008 at 7:32 PM | Permalink | Reply

    True comedy!

  5. Jeff Alberts
    Posted Sep 12, 2008 at 7:40 PM | Permalink | Reply

    Am I correct in assuming that in the first graph the black is Polar Urals, and the red is Yamal?

    • Posted Sep 12, 2008 at 7:54 PM | Permalink | Reply

      Re: Jeff Alberts (#5),

      Correct.

      Re: Jeff Alberts (#6),

      Because the “F word” has legal implications. I think it’s fairly safe to call it scientific misconduct, because frankly, ever since we’ve had the pleasure of analyzing Dr Mann’s work in climate science, he appears incapable of not massaging his data if it improves the “result”.

  6. Jeff Alberts
    Posted Sep 12, 2008 at 7:44 PM | Permalink | Reply

    I still can’t understand why you don’t call this what it is, Steve. You know what I mean, the word you snip out whenever mentioned. It’s so blatantly obvious.

  7. Craig Loehle
    Posted Sep 12, 2008 at 7:53 PM | Permalink | Reply

    Mann’s got to get up pretty early in the morning to outsmart our favorite auditor…

  8. John Baltutis
    Posted Sep 12, 2008 at 8:18 PM | Permalink | Reply

    Re: #3

    There’s a typo in the link I posted. This one should work:
    ftp://ftp.ncdc.noaa.gov/pub/data/paleo/contributions_by_author/mann2008

  9. Gerald Machnee
    Posted Sep 12, 2008 at 8:53 PM | Permalink | Reply

    There must be an expert somewhere who can say it is perfect.

  10. bill-tb
    Posted Sep 12, 2008 at 9:09 PM | Permalink | Reply

    You might be interested in this, it was specifically targeted at the type of ‘rabbit hunt’ that climate data has become …

    The U.S. has a law, Data Quality Act of 2001, this law requires federal agencies to ensure the integrity of the information they use and distribute. It also allows outside parties to petition to force the correction of information they believe is wrong.

    The Data Quality Act (DQA) passed through the United States Congress in Section 515 of the Consolidated Appropriations Act, 2001 (Pub.L. 106-554). The Government Accountability Office calls it the Information Quality Act, while others call it the Data Quality Act.

    The guidelines under subsection (a) shall –

    (1) apply to the sharing by Federal agencies of, and access to, information disseminated by Federal agencies; and

    (2) require that each Federal agency to which the guidelines apply –

    (A) issue guidelines ensuring and maximizing the quality, objectivity, utility and integrity of information (including statistical information) disseminated by the agency, by not later than 1 year after the date of issuance of the guidelines under subsection (a);

    (B) establish administrative mechanisms allowing affected persons to seek and obtain correction of information maintained and disseminated by the agency that does not comply with the guidelines issued under subsection (a); and

    (C) report periodically to the Director –

    (i) the number and nature of complaints received by the agency regarding the accuracy of information disseminated by the agency; and

    (ii) how such complaints were handled by the agency.

    This could be fun.

  11. Steve McIntyre
    Posted Sep 12, 2008 at 9:21 PM | Permalink | Reply

    Here’s another interesting data series in the Mann archive of “original” proxies at WDCP, one which shows a remarkably linear trend. Now this “proxy” wasn’t used. I wonder what criterion it failed.

  12. bernie
    Posted Sep 12, 2008 at 9:24 PM | Permalink | Reply

    #12 Steve
    Axis labels?

  13. Joel McDade
    Posted Sep 12, 2008 at 9:24 PM | Permalink | Reply

    #12 autocorrelation alert!

  14. bender
    Posted Sep 12, 2008 at 9:25 PM | Permalink | Reply

    Show us the error bars you incompetent hack!

  15. Dishman
    Posted Sep 12, 2008 at 11:03 PM | Permalink | Reply

    I’m confused now.

    Which data set did he actually use in his paper?

  16. Mark T
    Posted Sep 13, 2008 at 12:14 AM | Permalink | Reply

    How often do these things need to crop up before somebody cries shennanigans?

    Mark

  17. Louis Hissink
    Posted Sep 13, 2008 at 1:26 AM | Permalink | Reply

    #1

    John A

    snip

    There are other ways we can nail data biassing in the mining industry but academia is quite different as Steve shows.

    Mind you I am slack-jawed at the statistical massaging to the data – If I was allowed to use all of my data for the potential mine I am involved in, halleluya – but QAQC constraints limit what IO can use, and I actually have to do extra drilling to fill in the gaps in the data coverage – just to comply with JORC guidelines for transparency and accuracy.

    (I am on the AIG complaints committee so I have to be totally kosher in this area).
    snip

  18. Dodgy Geezer
    Posted Sep 13, 2008 at 2:04 AM | Permalink | Reply

    “How often do these things need to crop up before somebody cries shennanigans? – Mark”

    Perhaps a more useful question would be ‘To whom would one cry shennanigans?’.

    It has become obvious over the past years that there is no facility in climate science, as there is, for instance, in the financial or commercial world, for complaining to an independent tribunal. In the absence of such a body, any complaints raised on Climate Audit can be, and are being, ignored.

    Even if Dr Jolliffe makes a comment at Tamino’s, the Hockey-stick (Mark 52a) can still be published in Proc. Natl Acad. Sci., and welcomed in Nature. It may be disproven comprehensively in CA, but that is irrelevent. After all, it has been comprehensively PROVEN at RC, hasn’t it?

    We are all amazed at what Steve has managed to do for Science, working against all that the international establishment has thrown at him. But we need to find some way of levering this body of work, one of the few ‘proper’ climate science resources, into establishment view. Dr Jolliffe, for instance, should have known about these discussions 5 years ago….

  19. Demesure
    Posted Sep 13, 2008 at 2:52 AM | Permalink | Reply

    @12
    Incredible !
    With such crap “data”, Mann is claiming to reconstruct climate with 1357 “proxies”. And it’s archived and co-authors or reviewers have nothing to say about that.
    I’m without words to qualify such bllunder.

  20. Louis Hissink
    Posted Sep 13, 2008 at 3:44 AM | Permalink | Reply

    #20

    The problem is institutionalised science – government science in other words, and always politically directed. Like it or not.

    There isn’t a solution to it because while the commercial world can be forced to report to legistated, or in self governance in terms of the OZ JORC guidelines, mandated requirements, who is to police the police? Mann et al are government, as is Hansen etc.

    Politicans may come and go, but the infrastructure remains and there lies the problem and its solution.

    Despite all Steve’s efforts here, one fact is obvious – we, as individuals, do not control government let alone the unelected UN, and it is to this unelected body that the whole schmozzle can be placed.

    I sometimes get the feeling that we are being allowed to vent our spleens via the blogs to let off steam. Once deflated, and there any only so many hours in a day, one simply has not the energy to continue with this if one also has a fulltime job. The amount of energy need to refute crass stupidity is sometimes beyond comprehension – and faced with this fact, most of us then give up – and lose.

    We live in interesting times.

  21. Louis Hissink
    Posted Sep 13, 2008 at 3:50 AM | Permalink | Reply

    #22

    JohnA

    now now, cynicism is unbecoming :-)

  22. Louis Hissink
    Posted Sep 13, 2008 at 3:55 AM | Permalink | Reply

    Re #12

    Steve,

    Not a graduate student’s effort? Let’s remeber that academics generally don’t due their own data crunching – they assign that to the slave students and other academic menials – usually unsupervised as well, so all sorts of problems might arise.

    So is it Mann or his “Mexican Labour”, (aka students) who are fault????

  23. EW
    Posted Sep 13, 2008 at 4:14 AM | Permalink | Reply

    Slave students? I don’t know, but surely in the media high-profile hockey stick case Mann should cast an extra sharp eye on the process of data archiving, no?

    I archive my DNA data personally prior submitting the manuscript. I must admit that I also archived data that I later found as erroneous (mistakes in experiments happen, unfortunately). Then I myself had to clear the mess afterwards…and that is only a minor relatively unimportant area in mycology.

    • Posted Sep 13, 2008 at 4:12 PM | Permalink | Reply

      Re: EW (#26), I have yet to meet anyone that hasn’t made a mistake. It’s how you deal with them that makes the person. If Mann has two different data sets archived, a simple note would resolve the issue. Not seeing that simple note, like John Christy did and does, leads me to believe that Mann has a large hubris coefficient that may be auto correlated with a number of his peers. There are so many questionable and a few flatly wrong proxy reconstructions referenced by both sides that I feel this is more of a beauty pageant than a scientific pursuit. It would be nice if the PR Challenge was the real deal and academicians could learn to deal with more colloquial expressions without their undies bunching. It wouldn’t hurt Steve to chill just a touch in the interest of diplomacy, though I see his frustration.

      Back to lurking

  24. Louis Hissink
    Posted Sep 13, 2008 at 4:23 AM | Permalink | Reply

    #26

    Well put, but arrogance has its presumptions, and in the high pressure job of saving the world, who has the time to check all the minor details – I mean over 1000 proxies!

    But I do understand the problem with data archiving – lamentably employing simians, usually homo, cause data errors down the track which can’t be reversed. If it’s mundane data, fair enough, but iconic data forming the basis of the HT????

    Arrogance would be my conclusion.

  25. advance
    Posted Sep 13, 2008 at 6:18 AM | Permalink | Reply

    interesting article +1

  26. Gerald Machnee
    Posted Sep 13, 2008 at 7:01 AM | Permalink | Reply

    Was that withdrawn file “Censored”?

  27. Fred
    Posted Sep 13, 2008 at 8:47 AM | Permalink | Reply

    in related news, after years of playing with the data from Antarctica to “prove” the ice caps there are shrinking, NASA has now recanted. No word if the rack was needed to extract the confession. The state of denial is still in place as they claim teh expanding ice sheets are caused by . . . . ta da AGW.

    “The Antarctic wintertime ice extent increased…at a rate of 0.6% per decade” from 1979 to 2006, says Donald Cavalieri, a senior research scientist at the NASA Goddard Space Flight Center in Maryland. … Since 1979 … the average year-round ice extent has risen too.”
    http://tinyurl.com/4gktm6

    Steve: Why place this on this thread? Please look for a related thread.

  28. Posted Sep 13, 2008 at 11:01 AM | Permalink | Reply

    I have been trying to find the original data at the links above. But there are only 1209 data sets in them. Can someone point me in the right direction.

    Steve: http://www.ncdc.noaa.gov/paleo/pubs/mann2008/mann2008.html has the other 148 series in the proxy data.

  29. GeneII
    Posted Sep 13, 2008 at 12:01 PM | Permalink | Reply

    A quote here that applies :

    “Ninety nine percent of my life I was lied to.”
    ~~Slim Shady

  30. John Baltutis
    Posted Sep 13, 2008 at 2:37 PM | Permalink | Reply

    javascript:edInsertContent(edCanvas,’Re:%20%3Ca%20href=%22#comment-296620%22%3EJeff%20Id%20(%2332)%3C/a%3E,%20′);updateLivePreview();

    go to ftp://ftp.ncdc.noaa.gov/pub/data/paleo/contributions_by_author/mann2008 and download the proxy-original.zip file.

  31. Paddy
    Posted Sep 13, 2008 at 4:28 PM | Permalink | Reply

    How many federal agencies conducting climate research that is subject to the Data Quality Act have issued the guidelines and established the administrative mechanisms mandated by the Act? How many of these agencies report to the GAO regarding complaints received about mishandling data and how those complaints were resolved? How many complaints have been filed with GISS by interested readers of this blog about Hansen’s and Mann’s mishandling or failing to archive data?

    If NASA, NOAA and their departments or sub-agencies like GISS have not complied with DQA rules regarding issuing written guidelines and reporting, complaints should be made to GAO to force compliance.

    If no complaints have been submitted to GISS about improper data revision and failure to archive, what are you all waiting for? Force them to comply with their own rules and report their results to GAO. If GISS refuses or performs imperfectly, complain to GAO.

    From what I have read here and WUWT, there is ample justification to formally complain about DQA non-compliance. Force GISS to prove that it has and is complying with DQA.

    When you deal with a part of government you have to operate under its rules. Typically when there are illegal activities or non-compliance, remedies are available. I submit that GISS is ripe for review.

    I have retired, but I made a living keeping several federal agencies honest.
    I know how to obtain administrative remedies for agency abuses. The place to start may be getting advice from a public interest law firm like the Pacific Legal Foundation, whose mission is protect the public from government misconduct.

  32. J. Peden
    Posted Sep 13, 2008 at 4:44 PM | Permalink | Reply

    Within a day, on the afternoon of Sep 5, the data set was deleted and replaced with another data set,…

    If I were a player of Fantasy Football, and this happened, I would be very upset. Why can’t Mann even approach the standards of Fantasy Football?

  33. Posted Sep 13, 2008 at 5:27 PM | Permalink | Reply

    I know this will reveal the depth of my ignorance, but while I was able to successfully download the proxy-original.zip file, I cannot figure out what program I need to actually view the data sets which are in PPD format. Can some one help?

    • Sam Urbinto
      Posted Sep 15, 2008 at 12:32 PM | Permalink | Reply

      Re: sue (#39),

      They look like space or tab delimited text files to be imported into things like spreadsheets and math progams et al. Try changing the extensions to ccv or what have you.

      1.9880000e+003 1.0060000e+003 1.7000000e+001

  34. Lars Baath
    Posted Sep 14, 2008 at 9:16 AM | Permalink | Reply

    I have enjoyed your discussions here but have not followed everything so my posting here may be out of date. I do not see any estimates of error bars in any of the data files and have not seen any discussion or publication on this. To me it a very strange concept to add data from different proxies together without taken into account any kind of errors, individual and especially systematic.

  35. MarkR
    Posted Sep 14, 2008 at 10:07 AM | Permalink | Reply

    Paddy. I have said much the same thing in the past. There are rules, Federal, and Institutional, which are not being followed, and apparently (apart from Freedom of Information Act enquiries in the UK), no-one is trying to enforce them. Maybe you will be the one.

    In 1973, the American Statistical Association—Federal Statistics Users’ Conference Committee on the Integrity of Federal Statistics reported that:
    Nothing could undermine the politician and implementation of his policy recommendations as much as an accumulated and intense public distrust in the statistical basis for the decisions which the policy-maker must inevitably make, or in the figures by which the results of these decisions are measured. Unless definite action is taken to maintain public confidence in Federal statistics and in the system responsible for their production, there will be growing tendencies to distrust leadership.
    With respect to trust in the Federal statistical system, President George H.
    W. Bush stated in 1990:
    It is of paramount importance to this Administration that these fundamental principles of the Federal statistical system are strictly maintained so that the accuracy and integrity of Government data are not threatened.
    In 1995, the Congress reauthorized the Paperwork Reduction Act (PRA), which makes OMB responsible, among other requirements, for coordination of the Federal statistical system to ensure the integrity, objectivity, impartiality, utility, and confidentiality of information collected for statistical purposes.

    • Craig Loehle
      Posted Sep 14, 2008 at 10:34 AM | Permalink | Reply

      Re: MarkR (#41), I do not believe the data in question, either GISS or NASA GCMs or especially NSF studies would be classified as “government information collected for statistical purposes” in the sense that balance of trade data, unemployment, crime statistics etc. are. I would bet they are classified as “research”.

  36. Dishman
    Posted Sep 14, 2008 at 11:07 AM | Permalink | Reply

    GISTEMP (and all other temperature/ice extent/… data) should be “government information collected for statistical purposes”.

    I’ll ask again…

    Which data set collection did Mann actually use? The one he archived in the SI, or the one he archived with WDCP?

    • Ernie
      Posted Sep 14, 2008 at 3:55 PM | Permalink | Reply

      Re: Dishman (#43), That’s the time wasting problem isn’t it, you have to try both sets and see which replicates his results. If neither do then you cry foul.

      – Ernie.

  37. EW
    Posted Sep 14, 2008 at 11:43 AM | Permalink | Reply

    #336 (captdallas2)
    In my erroneous DNA data case, my pride and vanity wouldn’t survive someone other pointing that to me or criticizing it in their paper, so I’ve chosen to eat the humble pie instead, correct the archived data and write (with a very sour face) e-mails about the mistake and its correction to other researchers working on related fungi. Don’t know about Mann, but having someone like Steve McI behind my back would make me extra paranoid about what I’m going to archive or publish.

  38. A,Syme
    Posted Sep 14, 2008 at 4:29 PM | Permalink | Reply

    This whole business just gets stranger and stranger!

  39. Dave Salt
    Posted Sep 14, 2008 at 5:12 PM | Permalink | Reply

    For those who think the Hockey Stick is dead, you may be interested to know that the BBC broadcast the second in a series of three programmes tonight (14/09/08) on climate change, entitled “Earth – The Climate Wars” (BB2 9pm), in which Dr Ian Stewart pronounced that there was sufficient evidence from other sources, besides Mann, to conclude that it was real.

    The funny thing is that he also showed the controversial plot of solar activity versus time from the “Great Global Warming Swindle” with the missing temperature data from the end of the 20th century – which indicated a break-down of the correlation – but forgot to add any temperature data from 2000 onwards!

    I’ve already emailed a complaint to the BBC but don’t expect it will change anything.

    • Dodgy Geezer
      Posted Sep 15, 2008 at 7:42 AM | Permalink | Reply

      Re: Dave Salt (#47),

      I’ve already emailed a complaint to the BBC but don’t expect it will change anything…

      So you watched it, did you? I didn’t catch it, but I would like to add my complaints to yours, and would encourage anyone else to do the same. A volume of complaints from different people never hurts!

      Were there any particularly egregious points made? The BBC rules require balance and no obvious provable error, so you will need to make specific points about misrepresentation to get a response…

      • Dave Salt
        Posted Sep 15, 2008 at 2:40 PM | Permalink | Reply

        Re: Dodgy Geezer (#54),
        I’d need to view the programme again to be sure, but I don’t think there were any obvious/explicit errors. However, the general tone and manner in which key skeptic arguments were dismissed made me feel very uneasy about Dr Stewart’s attempt at impartiality. Anyway, just for the record, here’s what I wrote…

        “Dr Ian Stewart’s analysis of the current scientific “consensus” of the evidence for catastrophic man-made global warming had several serious omissions regarding:
        1) The underlying statistics that form the basis of the so-called “hockey stick” still remain very questionable and so cannot be regarded with the level of certainty that he presents (Cf. the work of Steve McIntyre);
        2) He complained that temperature measurements from the last decade of the 20th century were ignored in a plot of solar activity but, when showing a more complete plot, he then himself ignores the temperature trends of the last eight years, which show a distinct drop;
        3) His statement about there being now only a tiny minority of so-called skeptics completely ignores the list of scientists who have signed petitions like the Manhattan Declaration (Cf. link at http://www.climatescienceinternational.org) and others like the Petition Project (www.petitionproject.org).

        Based upon the above, along with the general tone of the presentation, Dr Stewart’s argument exhibits a clear bias that undermines any attempt to present a scientific assessment of this subject. This is indeed a pity because it would have been an ideal opportunity for the BBC to demonstrate its interest in scientific integrity and to dispel any suggestions of social or political bias.”

        There were other things, like the way he seemed to edit interviews with people like Roy Spencer that, to me, made them look foolish, but I guess this is somewhat subjective. I also don’t understand why he mentioned that William Herschel believed that the sun was inhabited, immediately after saying that Herschel was one of the first to believe that the sun was a influence on the Earth’s climate — maybe this was an attempt at humor that I just didn’t get?

        Having said that, the distinct lack of any mention of recent temperature trends did make me feel very uneasy. However, as the programme is intended to cover the arguments over the last three decades, I’ll give Dr Stewart the benefit of the doubt an assume he will cover this in the third and final part, next week, in which he will also discuss climate models.

  40. Posted Sep 14, 2008 at 6:31 PM | Permalink | Reply

    I don’t know about you, but I spent my Sunday plotting the missing, scrapped, tossed and lost data. You might find it interesting.

  41. Posted Sep 14, 2008 at 6:31 PM | Permalink | Reply

    The link didn’t work.

    http://noconsensus.wordpress.com/2008/09/15/mann-08-the-missing-data/

    • Pat Keating
      Posted Sep 15, 2008 at 6:26 AM | Permalink | Reply

      Re: Jeff Id (#49),

      That IS very interesting. I wonder why they were removed……;>)

  42. Edward T
    Posted Sep 15, 2008 at 1:33 AM | Permalink | Reply

    Would it be possible to put Mann’s gw machine online so we can plug in data and create our own ‘hockey sticks’?

  43. rhodeymark
    Posted Sep 15, 2008 at 6:23 AM | Permalink | Reply

    Does the study of Deleted Mann make one an anthropologist?

  44. Pete
    Posted Sep 15, 2008 at 6:29 AM | Permalink | Reply

    Edward T,

    Excellent idea. It could become an educational tool used in the intro to basic statistics (and journalism) so students get an appreciation of some pitfalls in improper use of statistical methods (or weak journalism research).

    In the near term, it should be provided to someone in the media like John Stossel.

    • Edward T
      Posted Sep 15, 2008 at 9:52 AM | Permalink | Reply

      Re: Pete (#53),

      It’s hard to explain the statistical gymnastics -=snip – because of the reason they are warmies: they prefer pictures of polar bears and and slogans 10 on the monbiot scale of environmental outrage to statistics, which they won’t understand even after patient verbal explanation.

      Also, accepting as true any climate-like data entered into Mann’s (there are 1000s other scientists!) bias correction (there is bias, prove it!) algorithm (now you’re playing semantics!) creates exceptional recent warming (look at the polar bears!), especially when the range of data is distilled down to graphs that could look like hockey sticks (the other data was bad, this is how science works!), would require the warmies treat their all-knowing heros as possible fraudsters (this is another ad hominin!).

      The truth bluntly put is an unacceptable humiliation for them. Give them a fun graph to play with – put in ‘red noise’ and lo! there’s a hockey stick – then they can email this to their friends and get the satisfaction of humiliating someone else and knowing the real truth first. Everyone loves a conspiracy.

      We should make this extraordinary, almost unbelievable episode in the history of climate science as easy to understand as possible – preferably in a fun, interactive way so everybody gets it.

  45. PeteW
    Posted Sep 15, 2008 at 7:47 AM | Permalink | Reply

    If you want to see it its available on BBC iPlayer

  46. harold
    Posted Sep 15, 2008 at 8:41 AM | Permalink | Reply

    Alas, you can only use the BBC iPlayer in the UK.
    Here you can hear Dr Iain Stewart talking about his Power of the Planet sreies and his *AGW conversion*:
    http://nl.youtube.com/watch?v=hrPjddK2t-I

  47. PhilH
    Posted Sep 15, 2008 at 10:12 AM | Permalink | Reply

    I am a long time lurker with no expertise in statistics but, as a retired judge, prosecutor and defense attorney who has seen a lot of strange stuff in the last forty years, I think I can comment on the social and cultural aspects of the “Team.” It occurs to me that an apt metaphorical description of their relationship is that it is “incestuous.” Somewhat less metaphorically, but just as apt, it also happens to describe what they are doing to each other.

    And, not incidentally, to the rest of us.

    • Ben
      Posted Sep 15, 2008 at 10:46 AM | Permalink | Reply

      Re: PhilH (#58), Wegman was the first authority to map this social network. It is worth noting that this paper has several co-authors who were NOT a part of that network.

  48. Steve McIntyre
    Posted Sep 15, 2008 at 10:57 AM | Permalink | Reply

    #59. Why, Ben, you’re absolutely on to something here. How could anyone claim that Mann’s students, Zhang and Miller, and Hughes’ associate, Ni, are not “independent” of Mann and Hughes?

    • Ben
      Posted Sep 15, 2008 at 11:14 AM | Permalink | Reply

      Re: Steve McIntyre (#60), Thank you. I find most students to be highly independent of their supervisors, and more than willing to prove them wrong given half a chance. Ababneh, for example.

      • JimR
        Posted Sep 15, 2008 at 2:14 PM | Permalink | Reply

        Re: Ben (#61),

        Although Ababneh’s dissertation didn’t agree with her adviser’s POV she wasn’t willing to openly disagree with Hughes or archive or provide her updated data. She still hasn’t publicly commented (as far as I aware). And her adviser (Hughes) was part of Mann 2008 and still didn’t use the Ababneh update.

  49. MarkR
    Posted Sep 15, 2008 at 11:48 AM | Permalink | Reply

    Craig #42. All the See, land and satellite data for the USA is collected via Federal Agencies, and under this proviso, all Federal Agencies have a duty to follow the detailed guidelines. For example all data and calculations must be available so they can can be replicated. Something which if enforced, would have stopped Mann and Hansen in their tracks years ago.

    • Craig Loehle
      Posted Sep 15, 2008 at 2:00 PM | Permalink | Reply

      Re: MarkR (#62), Sorry if I wasn’t clear. The field and satellite climate data are official and SHOULD be open to the public. We have seen with GISS how much resistance there has been to releasing the code and the fact that even now there is no documentation of how “adjustments” were done. However, research by scientists (even Hansen) are not considered in the same light, because research is by definition exploratory. There have been attempts under FOI to obtain notes/data of non-federal scientists funded by gov $, but not very successful and strongly resisted by academics.

      • MarkR
        Posted Sep 16, 2008 at 4:18 AM | Permalink | Reply

        Re: Craig Loehle (#65),

        However, research by scientists (even Hansen) are not considered in the same light, because research is by definition exploratory.

        Surely research, and published, peer reviewed papers are different? The latter are many times taken to be the final word, the definitive article.

        In addition. I believe that any and all Federal Institutions should insist that all information they publish should be derived from data and methods which are open to the public scrutiny. In this way, they would be following the law, and also (for example) the Hockey Stick could never be published as it is derived from data and methods which do not meet the legal criteria. Equally Hansen couldn’t publish his T charts, and his model forecasts, cos none of them have had their data and methods open to third party scrutiny, to establish their veracity.

  50. Posted Sep 15, 2008 at 12:41 PM | Permalink | Reply

    Ben, my experience in graduate school was quite the opposite. Students would bend over backward and forward to do whatever their major prof required of them. No degree without that signature. After graduation, the major prof becomes their source of professional contacts and the first node in their professional network.

  51. Sam Urbinto
    Posted Sep 15, 2008 at 2:22 PM | Permalink | Reply

    Oooops, I meant CSV (not ccv)

  52. Dave Clarke
    Posted Sep 19, 2008 at 1:57 PM | Permalink | Reply

    Steve says:

    The “wrong” data set – the one deleted from Mann’s website – is the one that Mann sent to WDCP, an archive of record, where Mann can’t change “inconvenient” data sets without leaving a trace. So WDCP has one “original” data set and Mann’s website has another.

    Steve, you and some of your readers seem to have problems with data sets and timestamps, so I’m going to help you out.

    I’m not sure if you noticed that *both* NOAA/WDCP “infilled” and “original” have 1357 series, whereas the SI has 1209 series (both original and infilled). Also notice that the timestamp of the files is Sep. 4, 15:14 in both the infilled and original WDCP archives. Also the file names don’t quite correspond.

    But this does mean that the 1357 “original” files were copied to the SI folder, archived, sent to the NOAA, then deleted and replaced with the 1209 “original” set actually used in the study. The end result is that the number of “infilled” and “original” data sets match up in each archive.

    So your scenario of Mann trying to change an “inconvenient” data set simply does not hold up. Why would Mann send the *two* larger sets to NOAA on Sep. 4 (or 5), if he was trying to cover up their existence? I find this highly implausible.

    As to why the SI contains only the set actually used in the study, while the NOAA contains a larger set, who knows. My guess would be that the 148 extra series were among those determined to be redundant or otherwise excluded as outlined in the SI.

    If I cared more about it, I would read all the available material very carefully, and then if I still had questions about the discrepancy, I would simply ask the authors before jumping to unwarranted conclusions.


    Steve:
    The WDCP 1357 file appears to be identical to the Sep 4 file that was deleted from Mann’s website. I did not “hypothesize” that Mann deleted the 1357 file from his website without a trace. That’s a fact. The disappearance of this file was documented and there is no change notice or change log at his site. It’s interesting that the infilled at WDCP is 1357 files as well. What a mess!

    As to why he would send the 1357 file to WDCP – I presume that it was done by mistake. If it wasn’t a mistake, then why did he delete it from his website? But really, it’s his job to explain what he’s doing in SI. I have not been able to get any information from him on other inquiries and do not expect that I would get an explanation of these machinations either.

  53. Steve McIntyre
    Posted Sep 19, 2008 at 7:01 PM | Permalink | Reply

    I’m not sure if you noticed that *both* NOAA/WDCP “infilled” and “original” have 1357 series, whereas the SI has 1209 series (both original and infilled). Also notice that the timestamp of the files is Sep. 4, 15:14 in both the infilled and original WDCP archives. Also the file names don’t quite correspond.

    It’s hard to keep up with all the different versions of Mann’s data. HOwever, your statement here is incorrect. The WDCP “original” dataset has 1357 series, but the WDCP “infilled” series has only 1355 series. There are more differences than it might appear at first sight.

    Both have 1209 series in common, but the 148 extra series in the WDCP original data set are different from the 146 extra series in the WDCP infilled series. The WDCP original data seems to correspond to the deleted PSU Sep 4 original (common timestamps as well). The WDCP infilled data corresponds to a deleted PSU Sep 4 infilled version that I happened to save.

    It has 146 “extra” series, which are 71 Luterbacher summer and 71 Luterbacher winter series, plus 4 log versions of the Lake Kortajarvi disturbed sediments.

    It looks as though Mann produced his “original” data on Sep 4 and then sent it to WDCP the same day. On Sep 5, he noticed some “problems” with his “original” data and deleted the Sep 4 “original” data from the PSU website, but seems to have been stuck with the Sep 4 version at WDCP.

    What a mess.

  54. Dave Clarke
    Posted Sep 22, 2008 at 1:22 PM | Permalink | Reply

    Steve,
    Sure, there is a discrepancy in the extra file sets between the two NOAA archived data sets (if you reread my comment you’ll see that I had noticed the file names of the extra data sets did not jibe, although I didn’t notice that there were two fewer proxies).

    Then you said:

    It looks as though Mann produced his “original” data on Sep 4 and then sent it to WDCP the same day. On Sep 5, he noticed some “problems” with his “original” data and deleted the Sep 4 “original” data from the PSU website, but seems to have been stuck with the Sep 4 version at WDCP.

    This is the real point: your insinuations about these file sets are not supported by the evidence. In fact, to me, it looks like the two NOAA archives were copied temporarily to the SI site for the purpose of archiving them. They were then deleted, but the zip files were only uploaded to the NOAA site days later.

    Look at the time stamp of the archives at NOAA and see for yourself:
    08/09/2008 4:48:00 PM

    So the scenario you have concocted where “inconvenient” file sets were mistakenly archived and couldn’t be withdrawn from NOAA site is clearly implausible. Of course, we still don’t know why they are different. But it is possible that the the two archives have a different purpose or requirement. Naturally at this point your relationship with Mann is strained by your constant accusations of bad faith, so it would be difficult to get what should be a relatively straightforward explanation.

    You also said:

    I’ll reserve comment for now on issues relating to the timestamp of this data set and the Gavin Schmidt hyperlink to it, presently pointing to a data version that did not exist at the time that the hyperlink was supposedly created.

    This is misleading. The data set that existed in the directory pointed to by Gavin’s SI link when it was “supposedly” created (high time you gave up on that insinuation, by the way) was a superset of the one that is there now, and contained all 1209 non-infilled proxy sets (along with 148 others). Have you found any differences at all in those 1209 proxies between the two “original proxies” archives?

    #6

    I still can’t understand why you don’t call this what it is, Steve. You know what I mean, the word you snip out whenever mentioned. It’s so blatantly obvious.

    Accusations – snip: forbidden word – , based on such flimsy “evidence” are utterly ridiculous.

  55. Steve McIntyre
    Posted Sep 23, 2008 at 1:17 PM | Permalink | Reply

    #72. Dave, in the quoted comment, I was not trying to be contentious, I was merely trying to guess as to the discrepancy between data versions – a guess that would obviously not be required if change logs were issued, as they ought to be. I stated:

    It looks as though Mann produced his “original” data on Sep 4 and then sent it to WDCP the same day. On Sep 5, he noticed some “problems” with his “original” data and deleted the Sep 4 “original” data from the PSU website, but seems to have been stuck with the Sep 4 version at WDCP.

    Given your doubts about this, I asked WDCP for information on when they uploaded the Mann SI and it was on Sep 4 as I surmised:

    Hi Steve,
    I downloaded the reconstructions and instrumental data files from
    Mann’s site: http://www.meteo.psu.edu/~mann/supplements/MultiproxyMeans07/
    on Sept. 4, and requested a .zip of the individual proxy files, which
    they produced on the 4th and I downloaded on the 5th. So yes, you are
    correct, we have archived files as of Sept. 4.
    -Bruce

    Dave, perhaps you could acknowledge that, in this case, my surmise was correct and that yours was not borne out by further information.

    As to the timing of Gavin’s inline response, I have reviewed the timing issues and will post on it sometime.

One Trackback

  1. [...] More on that hockey stick revival thing.  And Mann’s credibility. [...]

Post a Comment

Required fields are marked *

*
*

Follow

Get every new post delivered to your Inbox.

Join 3,119 other followers

%d bloggers like this: