Mannomatic smoothing: technical details

This post is rather technical, and it is intended mainly for the historical completeness. So unless you are very, very interested in the tiny technical details of the HS saga, you can safely skip this.

As most readers are aware, and stated in my post few hours after CG broke out, Mike’s Nature trick was first uncovered by UC here. He was able to replicate (visually perfectly) the smooths of MBH9x thereby showing that the smooths involved padding with the instrumental data. The filter used by UC was the zero-phase Butterworth filter (an IIR filter), which has been Mann’s favourite since at least 2003. However, there was something else that I felt was odd: UC’s emulation required a very long (100 samples or so) additional zero padding. So about two years ago, I decided to take an additional look at the topic with UC.

Indeed, after digitalizing Mann’s smooths we discovered that UC’s emulation was very, very good but not perfect. After a long research, and countless hours of experimenting (I won’t bore you with the details), we managed to figure out the “filter” used by Mann before Mann (2004)-era. Mann had made his own version of the Hamming filter (windowing method, an FIR filter)! Instead of using any kind of usual estimate for the filter order, which is usually estimated from the transition bandwidth (see, e.g., Mitra: Digital Signal Processing) and has typically the length of a few dozen coefficients at maximum, he used the filter length equal to the length of the signal to be filtered! As Mann’s PCA was apparently just a “modern” convention, this must be a “modern” filter design. Anyhow, no digital signal processing expert I consulted about the matter had ever seen anything like that.

In order to see how absurd the “filter design” is, consider filtering a signal of length 999 samples. According to Mann, you should design a Hamming filter of the same length. One should always disregard half of the filter length amount (i.e., 499 in our example) of filtered values from both ends, so in Mann’s case one would end up with a single smoothed value! In Mann’s implementation, however, one ends up with a filtered series of the same length as the original signal.

Another way to think Mann’s “filter” is to consider it as a normal filter with a huge (half the signal length) zero padding to the both ends of the signal. This interpretation also gives hints why UC’s emulation was so successful. One can also speculate, if the similarity of the results between zero-phase Butterworth and Mann’s original filters is the reason Mann chose the Butterworth filter in the first place.

If someone wants to explore this topic further, I’ve place my Octave/Matlab implementation of Mann’s smoother here. The code includes references to the original Mann’s code I uncovered. Finally, the exact parameters of the trick in MBH9x were as follows. MBH98 has 50-year smoothing with padding of 1981-1995 instrumental. Additionally, the smoothing is cut back 25 samples (half of the “filter length”) from both ends. MBH99 used 40-year filtering with 1981-1997 (not 1998!) instrumental padding. The smooth is cut back 20 samples from the end but not from the beginning.

Who wrote the EPA documents?

Jean S writes (transferred from a comment with the addition of a few headings):

A question for the experts: is it known who wrote and who were used as experts in the EPA documents? If not, is that information considered public (i.e., obtainable under FOIA or similar)?

The reason I ask is that I get very, very eerily feeling when reading certain parts of the EPA decision, especially this one. For instance, I think there are very, very few people in this world from whom the following paragraphs could originate (considering style, content and astonishing familiarity with Mann’s work): Continue reading

Misrepresentations and the Tainted Narrative of Mann’s Complaint

In a recent post, I observed that Mann’s Statement of Claim contained a bizarre misrepresentation about the nature of Mann’s research, as it falsely credited Mann with being “one of the first” to document the increase in 20th century temperatures. Reader PhilH, a retired judge, observed that, on its own, the misrepresentation was merely odd and that it would have significance for the pleadings only if it could be connected to the narrative of the case. In today’s post, I’ll try to do exactly that.

casey-at-the-bat-1888-grangerThe “money quote” chosen by Mann’s lawyers as supposed evidence of his “exoneration” by the “inquiries” is an EPA statement that manipulation of “temperature data and trends”  is a “myth”. This quotation connects to the paragraph 2 misrepresentation, but not to Mann’s actual research.  It’s a massive whiff by Mann’s lawyers. Continue reading

The “Discovery” of the EPA Inquiry

One of the essential elements in Mann’s reliance on EPA findings is his assertion that his supposed exoneration by EPA had been “widely available and commented” on in the media and had been “read by the Defendants”:

All of the above reports and publications were widely available and commented upon in the national and international media. All were read by the Defendants.

The claim that all nine “inquiry” reports had been “read” by Steyn and the other defendants is surely a fantasy on the part of Mann and his lawyers. While Steyn seems to be a man of eclectic interests, somehow I can’t picture him poring through the dreck of the turgid “reports” from the various inquiries.

I am particularly dubious of Mann’s claim that Steyn (and Simberg) had read the EPA documents. Some climate blogs took notice of the EPA decision denying various petitions for reconsideration of the EPA Endangerment Finding when it was issued, but none understood it to supposedly be an “investigation” and “exoneration” of Michael Mann, something which would have occasioned great interest in July 2010, then only a few weeks after the Muir Russell and second Penn State reports. The EPA denial decision was first included in a list of inquiries in an unmarked Feb 2011 revision of a November 2009 post by SKS, but really was brought to public attention for the first time in Mann’s Statement of Claim itself. It seems very improbable to me that Steyn (or Simberg) were aware of EPA’s supposed findings in connection in Mann (not that Mann’s characterization is accurate, but that’s a story for another day), or why they would be obligated to be familiar with them. Given Mann’s allegation that Steyn, Simberg and others were supposed to be aware of EPA’s investigation and “exoneration” of Mann, the contemporary unawareness of this supposed EPA investigation – especially at SKS and Real Climate – is really quite remarkable.

Continue reading

EPA on Mann’s “Fraud” Invective

Michael Mann, now feigning sensitivity towards Mark Steyn’s use of the word “fraudulent”, used identical language in the Climategate emails against critics without the slightest compunction. Mann’s hypocrisy has been widely noted.

Unpublicized thus far is a discussion by EPA, in which EPA concluded that Mann’s accusations of “scientific fraud” were within the scope of “acceptable and appropriate” scientific exchange and that it is “entirely acceptable and appropriate for scientists to express their opinions and challenge papers that they believe are scientifically flawed” in such terms.

The EPA’s finding appears to be inconsistent with Gavin Schmidt’s recent tweet arguing that such language is “per se defamatory”:

Saying that ppl [people] are frauds is per se defamatory. Goes beyond disagreement/error/dislike.

Continue reading

Spam

During the past few months, the volume of comment spam from spambots has increased by an order of magnitude. Right now, this blog receives about 1 spambot comment per minute. Akismet and other spambot blocking services do an excellent job and without such software, it would be impossible for blogs to function.

Nonethelss, I’ve noticed that the number of spambot comments being passed through has increased by an order of magnitude in the past few months. Anthony is also experiencing this. Because I moderate after the fact, spam comments appear here until I come online.

The blog software permits some customized screening. The word “blog” seems to appear in many spambot comments, so I’m going to try adding the word “blog” to my customized screening as this word. So avoid using this word in comments and if you do, expect it to be caught in moderation.

Yet Another Misrepresentation in the Mann Pleadings

In today’s post, I’ll discuss another misrepresentation in Mann’s Statement of Claim, one in which Mann bizarrely misrepresented the nature of his own research, falsely claiming credit for being “one of the first” to “document” the increase in 20th century temperatures. This particular false claim was in the same paragraph as Mann’s false claim to have received a Nobel prize. While the latter false claim has received widespread and well-deserved derision, Mann’s false description of the nature of his research has thus far passed without comment, an oversight that I will try to remedy in today’s post, which is part of a series of articles on various untrue statements by Mann in his pleadings in Mann v Steyn et al. For previous articles, see here. Continue reading

ACLU and National Media Intervene in Mann v Steyn et al

I get the sense that the Washington libel community and U.S. national media have belatedly woken up to the potential threat of Mann v Steyn and that the tide is now starting to run strongly against Mann in the anti-SLAPP proceedings. The most visible evidence of this is an impressive Amici brief from the ACLU and an imposing list of 25 other media organizations (the Reporters Committee for Press Freedom, the American Society of News Editors, the Association of American Publishers, the Association of Alternative Newsmedia (The Village Voice et al), NBC Universal, Bloomberg News, the publishers of USA Today, Time, The Washington Post, The Chicago Tribune, The Los Angeles Times, The Detroit Free Press, The Seattle Times, The Arizona Republic and The Bergen County Record) filed on August 11, 2014.

In addition, Steyn’s own Amicus brief substantially upped the ante on a separate front. It repeatedly and directly accused Mann of submitting “fraudulent” information to the court and commented adversely on “the ease with which Mann lies about things that would not withstand ten minutes of scrutiny in a courtroom”.

Other briefs are by CEI; National Review, the District of Columbia, the Alliance Defending Freedom; the Cato Institute, Goldwater Institute, Reason magazine and David Horowitz’s Individual Rights Foundation;, The Daily Caller, PJ Media, The New Criterion and various Internet publishers.

The Brief by ACLU and 25 Others
On some points, the ACLU brief takes a pretty similar line to that previously taken by National Review and CEI, but it presents some interesting fresh nuances and authorities. I won’t attempt to precis the brief as it is well written and worth reading, though readers uninterested in the procedural details of anti-SLAPP litigation would do well to skip to section II of the brief (commencing on page 12).

They asserted that the “challenged publications bear all the traditional hallmarks of opinion” and that “permitting such a defamation claim to proceed will substantially chill speech that challenges scientific conclusions, as well as public policies based on them.”

They observe:

Mann essentially complains that the defendants accused him of manipulating data, including by molesting and torturing it, to serve a political agenda. … Because the statements are quintessential opinions about the validity of Mann’s scientific methods and conclusions, they are entitled to full constitutional protection.

The ACLU brief provides detailed commentary on the range of abusive commentary in scientific disputes that courts have found to be permissible, concluding that the commentary in this case was within permissible limits.

In subsection II(D) (page 20), the ACLU et al provide an interesting and, in my opinion, compelling argument against Mann’s claim that the defendants were obliged to accept the findings of the various government agencies – a topic that I’ve focused on in my previous commentary.

CA readers are aware that Mann’s claim to have been “exonerated” by Muir Russell, Oxburgh, NOAA and the UK Government are untrue (“fraudulent” is Steyn’s term), but ACLU’s argument is different: they say that “punishing defendants’ speech because Mann’s work had been backed by other scientists or governmental agencies is contrary to core First Amendment principles”.

They first counsel the court against getting embroiled in trying to resolve scientific questions, no matter how authoritative the apparent support for the scientific assertion – advice that any court would undoubtedly be willing to heed.

The ACLU et al also unambiguously said that any reliance given by the Superior Court to supposed vindications or exonerations by governmental agencies was an “error” and that such government agencies did not give Mann the right to “silence his critics in a defamation claim”:

Furthermore, to the extent the Superior Court credited Mann’s assertion that investigations by the EPA, the National Science Foundation, and Penn State, among other scientific and governmental bodies, “laid to rest” defendants’ questions regarding Mann’s research, Am. Compl. ¶ 24, this too was in error. See also July 19, 2013 Orders at 16 (suggesting that statements were actionable because “Plaintiff’s work has been investigated and substantiated on numerous occasions”). The fact that certain official panels backed Mann’s methodology – facts that were not only disclosed in the challenged publications but in fact formed the basis for them – cannot allow him to silence his critics in a defamation claim. Under the First Amendment, the government is not the final arbiter of truth with the power to foreclose further challenge to its policies.11

In footnote 11, the Amici unambiguously supported defendants’ right to disagree with the findings of government inquiries, describing the right to such disagreement as fundamental to the First Amendment. In doing so, they observed that the defendants had provided substantive criticism of the procedures of the investigative bodies:

11 Indeed, if the First Amendment and case law interpreting it stand for anything, it must be that disagreement with findings of government and quasi-government bodies are fully protected. Here, defendants criticized the investigative bodies as, variously, lacking independence, failing to interview a relevant witness and therefore being too limited in scope, and being overly reliant on evidence provided by Mann’s employers, who had “so much at stake.” See Am. Compl., Ex. A.

In the next footnote, they pointed out that Mann’s pleadings had omitted the relevant information that the commentaries had disclosed that the government agencies had sided with Mann (while disagreeing with the agencies) and had provided hyperlinks to the criticized agency reports thereby permitting readers to form their own conclusions:

In this regard, Mann’s description of the commentaries omits that they disclose that governmental agencies had in fact sided with him while also criticizing those official findings. That background is described in their text – and, in some instances, through hyperlinked sources – thereby allowing readers to formulate their own judgments about the opinions expressed. See, e. g., Boley, supra, 950 F. Supp. 2d at 262 (hyperlinking to an earlier article provided “the necessary context for the allegedly defamatory remark”); Abbas, supra, 975 F. Supp. 2d at 18 & n. 7 (finding hyperlinks were sufficient to disclose background for fair comment privilege); Agora, Inc. v. Axxess, Inc., 90 F. Supp. 2d 697, 704-05 (D. Md. 2000) (dismissing defamation claim based on facts disclosed through hyperlinks), aff’d, 11 F. App’x 99 (4th Cir. 2001); Adelson v. Harris, 973 F. Supp. 2d 467, 483 (S. D. N. Y. 2013) (relying on a hyperlink to a report about an official proceeding in dismissing a defamation claim).

The ACLU brief closes:

At bottom, a participant in the “rough-and-tumble” of public debate should not be able to use a lawsuit like this to silence his critics, regardless of whether one agrees with Mann or defendants. See Guilford Transp. Indus., supra, 760 A. 2d at 595-96 (endorsing Voltaire’s philosophy, “‘I disapprove of what you say, but I will defend to the death your right to say it,'” which “anticipatorily articulated the spirit of our First Amendment”). The “law certainly does not insist” that a speaker “look kindly on [his] subjects,” nor that a plaintiff “simply by filing suit and crying ‘character assassination!,'” may “silence those who hold divergent views, no matter how adverse those views may be to plaintiffs’ interests.” Underwager v. Salter, 22 F. 3d 730, 736 (7th Cir. 1994). Rather, as the Seventh Circuit eloquently put it, expressing a sentiment echoed by other courts: “Scientific controversies must be settled by the methods of science rather than by the methods of litigation. More papers, more discussion, better data, and more satisfactory models – not larger awards of damages – mark the path toward superior understanding of the world around us.” Id. (citation omitted).13

Steyn’s Brief
Steyn’s brief (noted up together with the other briefs in Steyn’s blogpost here is far more direct in its allegations against Mann than previous pleadings, repeatedly describing claims and assertions in Mann’s previous pleadings as “fraudulent”.

Steyn rubbed salt into Mann’s false claim to have been a Nobel Prize winner and the imaginary tort of defaming a Nobel Prize winner, describing Mann’s priod claim as a “fraudulent misrepresentation”:

Mann’s fraudulent misrepresentation of his credentials and academic standing later earned him a rebuke from Geir Lundestad, director of the Nobel Institute in Oslo. One can well understand why the exposure of Mann’s fraudulent claim should cause him embarrassment but it should surely not justify resetting the procedural clock back to the beginning on this case, which is what in effect happened.

Steyn repeatedly used the term “fraudulent”, also using phrases like “the audacity of the falsehoods in Mann’s court pleadings is breathtaking” and that it was “deeply disturbing that a plaintiff should make such fraudulent claims in his legal pleadings”:

In his later court filings, Mann has made equally preposterous and objectively false claims. For example, Mann has claimed that he has been “exonerated” by such bodies as the University of East Anglia, the U.S. National Oceanic and Atmospheric Agency, and even by the government of the United Kingdom, none of which have investigated Dr Mann at all, never mind “exonerated” him.

The audacity of the falsehoods in Mann’s court pleadings is breathtaking. For example, on page 19 of his brief below dated January 18, 2013, he cites the international panel chaired by the eminent scientist Lord Oxburgh, FRS as one of the bodies that “exonerated” him, whereas on page 235 of Mann’s own book, The Hockey Stick and the Climate Wars , he states explicitly that “our own work did not fall within the remit of the committee, and the hockey stick was not mentioned in the report.” It is deeply disturbing that a plaintiff should make such fraudulent claims in his legal pleadings.

Steyn added:

It is clear from the ease with which Mann lies about things that would not withstand ten minutes of scrutiny in a courtroom that he has no intention of proceeding to trial.

Conclusion
All in all, it seems to me that Mann is facing far more formidable opposition than in any of the previous hearings, both from the ACLU et al intervention and from increasingly formidable defendant briefs. Mann presumably chose DC as a forum because he thought that it would favor him, but appears to have overlooked the possibility of intervention by the ACLU and national media. I can’t imagine that the DC Appeals Court will want the quixotic decision by J Combs-Greene (mostly adopted by J Weisberg) to represent the public face of DC libel law and I anticipate a different decision.

WG2 Misleads on Undernourishment Trend

Food security was one of the touchstone issues of the IPCC WG2 Summary for Policy Makers.

In its chapter on food security (Chapter 7), IPCC’s single statement on changes on undernourishment was that 75 million were added to the undernourishment rolls between 2003-5 and 2007:

FAO’s provisional estimates show that, in 2007, 75 million more people were added to the total number of undernourished relative to 2003–2005 (FAO, 2008); other studies report a lower number (Headey and Fan, 2010). More than enough food is currently produced per capita to feed the global population, yet about 870 million people remained hungry in 2012 (FAO et al., 2012).

However, yields have steadily increased for nearly all major crops and, according to the most recent FAO statistics, The State of Food Insecurity in the World 2013 published in September 2013, the number of undernourished [in developing countries] decreased by 17% between 1990-92 (995 million) and 2011-13 (827 million), as illustrated in their figure below. [The corresponding world decrease - see central points in 2013 online data is also 17% - from 1.015 billion in 1990-2 to 842 million in 2011-2013.]

fao 2013 undernourished
Figure 1. From FAO, The State of Food Insecurity 2013.

The FAO 2013 report does not directly show the difference between 2003-5 and 2007 undernourishment, but current online FAO data shows a decline in undernourishment of 44 million from 2003-5 (central point) to 2007, as opposed to the increase of 75 million reported by IPCC. [The online FAO data reconciles exactly to 2013 Annual Report Table 1 by comparing the central point in the three-year intervals to the annual figure in the online data.]

The glaring inconsistency between the most recent FAO numbers and the IPCC report raises some obvious questions.

Why the difference? Continue reading

Paper justifying AR4’s use of a uniform prior for estimating climate sensitivity shown to be faulty

This is a guest post by Nic Lewis.

 

In July 2004 the IPCC held a Working Group 1 (WG1) Workshop on climate sensitivity, as part of the work plan leading up to AR4. In one session, Myles Allen of Oxford university and a researcher in his group, David Frame, jointly gave a presentation entitled “Observational constraints and prior assumptions on climate sensitivity”. They developed the work presented into what became an influential paper, Frame et al 2005,[i] here, with Frame as lead author and Allen as senior author.

Frame and Allen pointed out that climate sensitivity studies could be – whether or not they explicitly were – couched in a Bayesian formulation. That formulation applies Bayes’ theorem to produce a posterior probability density function (PDF), from which best estimates and uncertainty ranges are derived. The posterior PDF represents, at each value for climate sensitivity (ECS), and of any other parameters (fixed but uncertain variables) being estimated, the product of the likelihood of the observations at that value and the “prior” for the uncertain parameters that is also required in Bayes’ theorem. Continue reading

Follow

Get every new post delivered to your Inbox.

Join 3,193 other followers