Readers of this blog may have noticed some chaffing back and forth between me and Tim Lambert. Anyone that’s followed the chaffing may have noticed that Lambert has spent a lot of time criticizing studies by John Lott on guns on statistical grounds. On a personal basis, I dislike/hate guns and cannot imagine why anyone would want one in their house. I have pretty typical urban Canadian views. Intuitively, I’d be inclined to think that anyone purporting to prove that more guns leads to less crime is likely engaging in pretty suspect statistics and would be inclined to think that Lambert’s criticisms of Lott are probably meritorious. But it’s not a topic that so far has interested me enough even to read Lambert’s criticisms of Lott.
By chance, while I was googling a complete different statistical topic, I stumbled across another statistical take on Lott, also severely critical. What intrigued me is that these criticisms of Lott’s methodology parallel my criticisms of Mann’s methodology. Maybe I’ll have to look at the Lott criticisms some more.
Here’s the article that I noticed together with a quote:
Lott’s work is an example of statistical one-upmanship. He has more data and a more complex analysis than anyone else studying the topic. He demands that anyone who wants to challenge his arguments become immersed in a very complex statistical debate, based on computations so difficult that they cannot be done with ordinary desktop computers. He challenges anyone who disagrees with him to download his data set and redo his calculations, but most social scientists do not think it worth their while to replicate studies using methods that have repeatedly failed. Most gun control researchers simply brushed off Lott and Mustard’s claims and went on with their work. Two highly respected criminal justice researchers, Frank Zimring and Gordon Hawkins (1997) wrote an article explaining that:" just as Messrs. Lott and Mustard can, with one model of the determinants of homicide, produce statistical residuals suggesting that ‘shall issue’ laws reduce homicide, we expect that a determined econometrician can produce a treatment of the same historical periods with different models and opposite effects. Econometric modeling is a double-edged sword in its capacity to facilitate statistical findings to warm the hearts of true believers of any stripe."
Zimring and Hawkins were right. Within a year, two determined econometricians, Dan Black and Daniel Nagin (1998) published a study showing that if they changed the statistical model a little bit, or applied it to different segments of the data, Lott and Mustard’s findings disappeared. Black and Nagin found that when Florida was removed from the sample there was "no detectable impact of the right-to-carry laws on the rate of murder and rape." They concluded that "inference based on the Lott and Mustard model is inappropriate, and their results cannot be used responsibly to formulate public policy."
The analogy would be even more complete if Lott, in addition to a complicated methodology, did not disclose his data (until his feet were gradually and increasingly put into the fire), misrepresented his methodology in important particulars, failed to disclose adverse cross-validation statistics, etc. etc. When I see a comment about the impact of slight changes in Lott’s statistical model, the line of analysis seems pretty similar to our analysis of the presence/absence of bristlecones.
It may be that Lambert on Lott and M&M on Mann have more in common than we may have appreciated. Also, it’s pretty easy to understand that if you had some other studies by Lott’s co-authors finding similar conclusions, anti-Lottians would not necessarily be very impressed and would be quick to examine each one of those other studies to see exactly where the flaw was. I’ll try to spend some time working through the analogy.
Climate Audit 
47 Comments
from above:
‘Black and Nagin found that when Florida was removed from the sample there was “no detectable impact of the right-to-carry laws on the rate of murder and rape.”‘
Since Florida was the first state to pass a “shall issue” law, doesn’t the above statement agree with Lott’s original conclusions?
that is in order to observe any difference one would have to include Florida since that was the only state that changed?
I don’t want to get involved in nuances of Lott i.e. for the purposes of arguing the impact of bristlecones on MBH98, I don’t RELY on the impact of Florida on Lott or any other aspect of the criticism of Lott – obviously our critique of Mann is completely independent. In the Mann context, it’s obviously inappropriate that his results depend on the presence/absence of bristlecones and equally inappropriate that these results should not have been reported and equally inappropriate that he should have said that his results were robust to the presence/absence of dendroclimatic indicators, when he knew that they were not robust to the presence/absence of bristlecones. Imagine comparable circumstances in Lott (or Lott withholding cross-validation R2 statistics or the equivalent.)
I will venture the guess that more people have strong feelings, one way or the other, about guns than about statistical analyses.
I will venture another guess: this thread may get deluged with comments that have little to do with the analogy that Steve is considering.
I hope that second guess is mistaken.
I agree. This is not the place to discuss validity of an independent issue.
But it does show how statistical results can be drastically altered through inclusion/omission of a small piece of data.
I strongly feel that M&M have uncovered a fatal flaw in the analysis of MBH98, and they should be commended.
What is the status with Barton’s committee?
Actaully Steve, I think you should think twice about slagging Lott in such simplistic terms. That…guy from Rutgers is basically taking a crap all over the notion you have been pushing: openness and sharing data. My understanding is that in at least one area Lott has been pretty good–sharing his data and methods. Isn’t that what you want from Mann, et. al.? One could also argue that Lott’s issues with Lambert and others stem in large part from this openness. Lambert though tends to take it to a level that is…well bordering on the disturbed. You can see it with his stuff on your co-author McKitrick.
I also caution you to rethink the possible thought that this will get Lambert to let up on you. It wont.
My final advice is simple: Stay on target. Getting dragged off into guns and crime is going to divert your efforts from where they should be. Save the stuff like guns and crime for later.
Steve V., you’re undoubtedly right. I’ve tried to keep comments narrowly-based. I’ve been surprised at the market for the narrow focus of the blog – it was over 125,000 hits in July – so I’m obviously hitting some kind of audience without straying off point. The trouble with feeding blogs is that it’s easy to post up stuff off the top of your head and I guess I’ve done that here. The thought that I had in mind was that anyone slagging Lott for his methodology should be able to see worse methodology in Mann. To the extent that Lott was open, then Mann was that much worse than Lott. I certainly didn’t mean to vary any of my positions on openness or get into a debate on Lott’s methodology. So people: I’ve got a couple of choices here. If it looks like we’re going to debate guns, I’ll either have to freeze the comments on this post or delete the post entirely as an off-topic. I don’t want to support an after-market arguing back and forth on guns, so I may do the latter and post up a notice on this on another post.
Perhaps a better example than “Lambert, Lott, and Mann” would be Legates, Labat, and Mann. In this case, Legates et al. show that Labat et al.’s claim that global runoff increases with global temperature is a statistical artifact based upon a single data point, which, if removed, leaves no statistical relationship.
The title of paper and its abstract:
From Advances in Water Resources, Article in Press. “Comments on “Evidence for global runoff increase related to climate warming”” by Labat et al.
By: David R. Legates a), Harry F. Lins b) and Gregory J. McCabe c) a) Center for Climatic Research, University of Delaware, Newark, DE 19716-2541, USA b) U.S. Geological Survey, Reston, VA 20192-0002, USA c) U.S. Geological Survey, Denver, CO 80225-0046, USA
Abstract
We have examined the evidence presented by Labat et al. and found that (1) their claims for a 4% increase in global runoff arising from a 1 C increase in air temperature and (2) that their article provides the “first experimental data-based evidence demonstrating the link between the global warming and the intensification of the global hydrological cycle” are not supported by the data presented. Our conclusions are based on the facts that (1) their discharge records exhibit non-climatic influences and trends, (2) their work cannot refute previous studies finding no relation between air temperature and runoff, (3) their conclusions cannot explain relations before 1925, and (4) the statistical significance of their results hinges on a single data point that exerts undue influence on the slope of the regression line. We argue that Labat et al. have not provided sufficient evidence to support their claim for having detected increases in global runoff resulting from climate warming.
Well don’t get me wrong Steve M. I think Lott has done some rather questionable things at best. On the other hand you have much bigger fish on the hook that Lott. I say cut bait on Lott and stick with the big one you’ve got. Freezing the thread is the best, IMO. Deleting is not usually a good practice since it makes people what you might be trying to hide. So keep it and close the comments.
And beware Lambert; he is like a tarbaby in that once you get sucked in it is hard to extricate yourself. He will write dozens of posts often times on the most obscure things. He’ll pick apart every little mistake (Ah-ha! A dangling particple–I’ve got that McIntyre NOW!) and then extrapolate from there so that everything you do is tainted. Just ignore him, or at best don’t respond directly to him.
I don’t know. I think there are plenty of supporters of Steve who can keep HLH (Hunter Lambert Hearndon) occupied if need be. It’s not like any of the three were making substantive points which would require advanced math to counter. Would that they were!
BTW on the putative subject of this thread, I can’t speak to Lott’s statistics, but at least he didn’t make things up out of whole cloth like Bell-whatshisname did.
While I have had some inclination to mention some analogies between the Bell-whatshisname case and some MBH98 developments, I think of the
1800+ posts at HNN on the former, and would not want to encourage such a distraction here.
Lott did make things up like Bellesiles did. The parallels are kind of eerie. (Lott/Bellesiles) claimed that (brandishing a gun stops crime 98% of the time/gun ownership rates in the early republic were very low) based on (a survey/a study of probate records) he conducted. Unfortunately no-one can check (Lott’s/Bellesiles’) work because he claims that the data was lost in a (disk crash/flood) and indeed there isn’t any good evidence that the (survey/study) was ever conducted at all. You can find out more details from James Lindgren’s (report/review).
A Steve Sailer reader made an interesting comment a while back about the Lott/Bellesiles affair that may also bear on MBH98 vs MM05 controversy:
I’m getting the idea that replicating results may not be so routine in the physical sciences, either.
There is a LOT of bad science and a lot of bad papers.
Steve,
I concur with the other commentators (especially Steve Verdon). Don’t go off message and give some disturbed people another avenue to attack your work based on your motives.
I reommend locking and deleting this post.
re#14
With all due respect, John, I don’t see anything untoward happening in this thread. If anything the discussion has been trending toward a discussion of due diligence and fact checking, things which are precisely what this site is about. The only one who’s even brought up gun use per se is Steve himself.
It seems clear that my guess about a possible deluge was a mistake. Good!
re: 8, “And beware Lambert”
1) Dr. Lambert has his own problems, from when he ventured out of his expertise into thermodynamics and the cos vs. cos^2 computational error of MBH. On the latter, he has both reversed (corrected) and then revised his original position on this subject — both a clear lesson in checking carefully before jumping stridently out of one’s expertise.
2) Dr. Lott is now commenting on GW — IMO, this can only lead to an defocusing of his (Gun Policy, Cause and Effect) expertise, already under serious and arguably credible attack.
While I can see the observed double standard of Dr. Lambert’s methodological scrutiny between MBH and Lott (Mr. McIntyre’s point), the important point is flawed AGW paper(s) and how subsequent climate science and national/world policy is still under its grip. IMO, it is still a revisionist’s view to state that MBH ’98 and ’99 played a minor part in the IPCC’01 TAR and the present decade policy discussions… especially since current governments (see thread “Spot the Hockey Stick”) still use this icon (and the IPCC’01 TAR) to prescribe cost-ineffective policy, from its dramatic and insubstantiated (if not incorrect) form!
Correction:
“still use this icon (and the IPCC’01 TAR) to prescribe WHAT IS LIKELY TO BE cost-ineffective policy”
correction 17: “cos vs. cos^2” should be “cos vs. cos^.5”
all those guns helped when the Wolverines took on the commies (note that the Europeans abandoned us except for the Brits…)
Hye
I see that you are talking about one of my article in
advances in water resources but only of the comment of Legates.
I must say that i reply to all the statistical issue raised
by legates et al. also in ADWR and their all their issues
HAVE BEEN CLEARLY SHOWN AS WRONG WITHOUT DOUBT
However, you can check by downloading the global runoff data set describes in Labat et al. 2004 at the following web page
http://www.lmtg.obs-mip.fr/user/labat/ (see on the left global runoff data set)
Thanks for your attention
Best regards
D. LABAT
Hye
I see that you are talking about one of my article in
advances in water resources but only of the comment of Legates.
I must say that i reply to all the statistical issue raised
by legates et al. also in ADWR and their all their issues
HAVE BEEN CLEARLY SHOWN AS WRONG WITHOUT DOUBT
However, you can check by downloading the global runoff data set describes in Labat et al. 2004 at the following web page
http://www.lmtg.obs-mip.fr/user/labat/ (see on the left global runoff data set)
Thanks for your attention
Best regards
D. LABAT
David (Labat), thank you for the invitation to download the dataset. Unfortunately, there is no dataset there … when you click on the link to
ftp://ftp.lmtg.obs-mip.fr/foo/labat/global_runoff_dataset.xls
it can’t find it. In fact, there is no “labat” folder in the “foo” directory.
Je m’suis desolee …
w.
Hi Willis,
I think Dr. Labat just mistyped the link, it’s here, (with the data link on the right).
Will be interested to see what you make of it.
Oops, I see now with the corrected link that the data is not there. Dr. Labat?
Here is the message I get from Dr. Labat’s site:
Safari can’t open the page “ftp://ftp.lmtg.obs-mip.fr/foo/labat/global_runoff_dataset.xls” because it could not load any data from this location.
I have also emailed Dr. Labat directly about the situation, we’ll see what transpires. The odd part to me is that not only is the file not there, the “labat” folder in the “foo” directory doesn’t exist.
w.
Dear Willis Eschenbach
Thanks for your attention for my work.
The ftp server was completely deleted without my approbation.
In order to avoid previous issues, i paste the data at the bottom
of the page so you can now download freely the data
at the following page
http://www.lmtg.obs-mip.fr/index.php?option=com_lmtg_myhomepage&lmtg_subject=labat
with the link on the left
All my aplogizes.
Best regards
D. LABAT
Dear Dr. Labat:
Thank you for posting your data. Would that all climate scientists were as prompt and gracious as you.
(A note to folks who don’t use commas as thousands separators, don’t try to paste it into Excel. Paste it into a word processor and change the commas to decimal points).
A few initial questions from a first look at the data:
1) Your “Global” figure is not the sum of the continental figures. I can only assume that this difference is the runoff from Greenland, Australia, and Antarctica. Is this assumption correct, or is it an error in the transcription of the data?
2) If this is correct, how did you obtain or estimate this figure?
3) Why does it vary so widely? The standard deviation of the continental annual runoff figures (Asia, Africa, Europe, North and South America) are all between 8-10% of their mean. The standard deviation of their sum is 4% of its mean, in the range of what we’d expect. The standard deviation of the “Global” figure is 5% of the mean. But what I am calling the “Other” category (the difference between the continental totals and the “Global” figure) has a standard deviation of 29% of its mean. Why is that?
4) Finally, each of the continents is positively correlated with at least one (Africa and Europe) or two (Asia, North and South America) other contents. The “Other” category is negatively correlated with all of them.
All the best,
w.
The data set is posted up in an annoying format. Here’s a R script to produce a cleaned-up version of the table – lots of little ad hoc devices to wade through the html tags. The gsub command replaces commas with a period for use in an as.numeric command.
url=”http://www.lmtg.obs-mip.fr/index.php?option=com_lmtg_myhomepage&lmtg_subject=labat&lmtg_item=1″
test=scan(url,skip=336,sep=”\t”,what=””)
test=test[!(test==””)]
temp=(nchar(test)<15);test=test[!temp]
test=substr(test,16,nchar(test))
temp=as.numeric(substr(test,1,1))
test=test[!is.na(temp)]
test=test[1:714]
n=nchar(test)
test=substr(test,1,n-5)
test=gsub(“,”,”.”,test)
test=as.numeric(test)
test=t( array(test,dim=c(6,714/6) ) )
dimnames(test)[[2]]=c(“Africa”,”Asia”,”Europe”,”NorthAm”,”SouthAm”,”Global”)
test=ts(test,start=1879)
labat=test
plot.ts(labat) #this yields a plot of the data
If I understand well, Dr Labbat’s results, assuming they have some significance thanks to magical wavelets and miracle “reference” correlations capable of recovering missing data, killing spurious trends due to water usage change or river geo-engineering, show that there is MORE water on continents as a direct consequence of global warming ?
If that’s it, so his introductory
rhetoricsremark “Climate change is one of the major challenge that humanity will have to face over the next decades” contradicts the results of his research.Hye
i will not reply to “demesure” who is not able
to distinguish my name from a famous bier ….
concerning Willis Eschenbach remarks, all answers are included
in the reference labat et al. (2004). The main answer
could be that all results have been normalised to
baumgartner and Reichel (1975) estimates and that a 1/0.89
coefficient have been applied to global runoff data in order
to take into account Australia runoff.
Best regards
PS : Sorry for the format of the data but i try to answer
as quickly as possible …
May I suggest that you simply archive an ASCII file. Given that there’s a little interest in the issue, why don’t you post up a pdf of your reply to Legates?
#31. Your refusal to reply to demesure because he supposedly confused your name with a famous beer is both petty and wrong. As a Canadian, I can authoritatively say that the beer is spelled “Labatt” and demesure’s “Labbat” is neither. Given that you criticize Demesure for spelling, I presume that you mean the famous “beer”; I am unaware of any famous “biers”, although funeral directors may know of some.
After a huge amount of effort which took me about 27.500 ms (copy, paste, replace “,” by “.”, save), I end up with an ascii archive: Labat Runoff
Exhausting. Must be thinking to get down a double malt Labbatt Belgian beer.
Re 33: With regard to “bier”, I kinda’ like Kronenbourg. . .
Also, I’d like to thank David Labat, who appears to be French, for going to the trouble of answering in English.
Re: #33 Steve,
I suspect there are a number of famous “biers” including those of Lenin, and Mao and King Tut, the last of which I made a pilgrimage to in Cairo a dozen or so years ago.
Dear Dr. Labat:
Thank you for the explanation in your post. You say:
I fear that the scope of my research is too broad (and the scope of my wallet is too narrow) to pay $30 for every study of interest, much as I would like to read yours, so instead I have taken another look at your data.
There is a very clear offset of one year between the sum of your continental data (which I will call “Sum”) and the global data of Baumgartner and Richel (“Global”). This is most evident in 1877, when the difference between the Sum and Global data is only 1% of the larger … extremely unlikely. Shifting the Global data back by one year (or your data forward by one year) improves the residual (reduces the standard deviation of the residual by 8%), significantly increases the r^2 between the two datasets from 0.54 to 0.60, and brings the 1877 difference back to 10%, about where we’d expect it to be. I’d say the error is real, and needs to be fixed.
I have to confess, when I see this kind of error in a dataset, I get nervous … it makes me wonder about the problems I haven’t seen …
I was also surprised to see that there is only a tiny and statistically insignificant trend in both the Sum (+0.2% ± 2.4% per century) and Global runoff data (+0.4% ± 2.1% per century). Since there is a bit of a trend in temperature over that time (~ 0.4°C/century), I found this odd. Nor is there any clear relationship with HadCRUT3 temperature data, the r^2 in both cases is less than 0.00.
Finally, since the 95% confidence interval for the trend of both the Sum and Global data is greater than ±4% per century, it would seem difficult to me to tease out some statistical relationship between either dataset and any other variable. I suspect that your conclusions are correct, that precipitation in fact goes up as the world warms. It seems inevitable because there must be more evaporation in a warmer world, which has to go somewhere. However, proving that from the data you have shown seems difficult at best. Not impossible … but certainly difficult.
My very best to you,
w.
Hye
I am very pleased about your careful analysis.
Please read my artcile carefully and you will
see that my trend only ocurs above the 1925-1994 period.
Then, Baumgartner and Riechel study has benn published in 1975
so i do not understand your graphic.
Please make the same correlations as me and then we can talk.
Finally, I encourage of course all hydrologists
around the world to collect more and more data in order to make
my estimate better and better …
desole pour mon anglais mais j’avoue que je ne suis pas
a l’aise avec la langue de shakespeare …
Best regards
Hye
I am very pleased about your careful analysis.
Please read my article carefully and you will
see that my trend only deals with the 1925-1994 period.
Then, Baumgartner and Riechel study has been published in 1975
so i do not understand your graphic since this pulbication only gives
estimation of mean annual continental runoff.
Please make the same correlations as me and then we can discuss.
Finally, I encourage of course all hydrologists
around the world to collect more and more data in order to make
my estimate better and better …
desole pour mon anglais mais j’avoue que je ne suis pas
a l’aise avec la langue de shakespeare …
English is language of scientists and French language of love …
Best regards
and sorry demesure for your exhausting work
D. LABAT
Et moi, je ne suis pas a l’aise avec la langue de Moliere!
I’m not sure I’ll grant you that, but I will admit that, for the most part, French cheese is better than American cheese. 🙂
#40 Margo, speaking of American cheese (than which not just French cheese but any food is superior), an old joke about Canada ran along the lines that the Founding Fathers wanted to achieve a country with French culture, British know-how and American efficiency. Instead we got French know-how, American culture and British efficiency.
One of my early criticisms of MBH98 – and they’ve still not confessed – had to do with hydrology and France.
Mann allocated a precipitation series from Paris to a New England gridcell; one from Toulouse to South Carolina and one from Marseilles to Spain. I never quite understood his fascination with French precipitation series as proxies for global temperature with remarkable teleconnections. I used the phrase: The rain in Maine falls mainly in the Seine. In the Nature Corrigendum, he admitted that the series did not come from the cited source; instead said that they came from “NOAA” with no further identification.
#41 You must be thinking of “Kraft American Cheese Product Singles” when you write so disparagingly of American cheese, Steve. Kraft “American Cheese” isn’t cheese. It’s not even allowed to be called cheese by law, hence the “cheese product.” That said, there are plenty of excellent cheeses made below the 48th parallel. As a vegetarian, I’ve sampled a number of them. I agree, though, that the very best heart-stopping cheese is made in France, usually containing about 80% butter fat.
Some time back, I ran across a cheese imported from Israel, made in Nazareth. ‘Cheeses of Nazareth’ has had quite a run in our family humor since then. 🙂
Pat, remember the line from Monty Python’s “Life of Brian” (when they were standing too far back at the Sermon on the Mount to hear well) — “Blessed are the cheesemakers”…
That connection escaped me, Curt, thanks. 🙂 Rest assured it’ll be properly exploited from now on.
Which paper is it that attributes Bombay, India precip to what looks like Philly, PA?
MBH98. I really don’t know where the Bombay series came from. A couple of years ago, I calculated correlations of MBH Bombay to historical precipitation series and there was virtually no correlation to actual Bombay and the closest as I recall was Philadelphia, but it wasn’t high enough to prove a match (whereas Paris and Toulouse could be conclusively related to mis-located North American series.)
M&M to Mann: