As some readers have observed, there is a lively controversy regarding an influential recent paper by Reinhart and Rogoff. Herndon et al (of Raymond Bradley’s UMass-Amhertst) concluded that RR’s conclusions depended on a bad weighting method, inexplicable exclusion of data from certain countries and years and even an Excel coding error. All the sorts of issues that are familiar with Mann and the Hockey Team. Even the defences from Reinhart and Rogoff are eerily similar: no errors ever seem to “matter” because of some new and unfisked study. One big difference: Reinhart and Rogoff at least conceded things that were unarguable, whereas Mann and the Hockey Team concede nothing, not even things as incontrovertible as upside down use of contaminated data.
.
Without dwelling on the underlying economic issue, the parallels to the Hockey Team are unmistakable. A critique stated that their conclusions depended on three errors:
a bad weighting method, inexplicable exclusion of data from certain countries and years, and an Excel coding error.
Precisely the sort of problem that has been discussed at Climate Audit.
The defences from Reinhart and Rogoff are also identical to Hockey Team defences. Zeitlin states:
Yesterday’s defenses from Reinhart and Rogoff are most notable for what they do not contain: a defense of the methodology that Herndon and his colleagues criticize.
Instead, like the Team, in response to criticism of any given article, they point to some other article – generally one just hot off the press which hasn’t been fisked yet.
The business writers are less forgiving than environmental writers. Zeitlin observes:
This isn’t a very satisfying defense, for two reasons.
One is the simple sloppiness of the apparent error that Reinhart and Rogoff made in calculating the average GDP growth for high-debt countries. Barry Ritholtz put it well: “Whenever you hear the phrase ‘the fundamental case is intact,’ you best run screaming from the room.” It’s reminiscent of the “fake but accurate” defense of the forged memos about President George W. Bush’s service in the Texas Air National Guard.
The Reinhart-Rogoff error also seems to have depended on small datasets with puzzling exclusions. Zeitlin:
Dean Baker has a good breakdown of what went wrong. What happened is that Reinhart and Rogoff included data from New Zealand in 1951, a year when debt/GDP exceeded 90 percent and GDP growth was -7.6 percent. But they excluded New Zealand data for the period 1946 to 1949, when debt also exceeded 90 percent and growth was often very strong. Include those years, and New Zealand’s average economic growth in high debt years is not -7.6 percent but +2.6 percent. Because there were only seven countries in the data set that Reinhart and Rogoff used to calculate average GDP growth under high debt conditions, and because they weighted each country’s average growth equally, getting New Zealand wrong by more than 10 percentage points was a very big deal, shaving 1.5 percentage points off their estimate of average growth.
The style of analysis in Reinhart and Rogoff is also very similar to the style of analysis in paleoclimate: relatively small data sets. In both cases, the analysis is statistical, but not exotic statistics.
It turns out that much of the resonance of the original Reinhart-Rogoff work depended on a compelling graphic: their “cliff chart”. Zeitlin:
Unlike the cliff chart that made the Reinhart-Rogoff paper so arresting, this way of looking at the data suggests that the historical performance of countries with varying debt-to-GDP ratios has little to tell us about what today’s fiscal policies should be.
Critics also observe that the authors have been highly involved in advocacy:
Reinhart and especially Rogoff have spent years now engaged in a high-profile political advocacy campaign grounded in a causal interpretation of their empirical work that both of them knew perfectly well was not in fact supported by their analysis
Maybe Herndon and coauthors will now take in an interest in work by their UMass colleague, Raymond Bradley.
Climate Audit 
73 Comments
Please do not discuss economic issues relating to whether RR were “right” or not. I will snip or delete. Only methodological points,
I noticed also that one of the authors (either Reinhart or Rogoff) also came back with the same retort: “It doesn’t significantly change the point of the paper”.
Steve: yup. the language is very similar.
Maybe because it doesn’t:
http://blogs.wsj.com/economics/2013/04/17/reinhart-rogoff-admit-excel-mistake-rebut-other-critiques/
Steve – Ivan’s link to the WSJ article, which is a response from Reinhart & Rogoff, deserves to be highlighted within the main article.
It would be far better to examine what Reinhart & Rogoff actually say in defence than to quote what “Zeitlin” says they say.
It’s also worth mentioning that the articles you link to don’t include the quotes you attribute to Zeitlin. I think you mean this article by Josh Barro, not Zeitlin: http://www.bloomberg.com/news/2013-04-17/reinhart-rogoff-on-debt-and-growth-fake-but-accurate-.html.
The quotes in the Barro article (“Yesterday’s defenses from Reinhart and Rogoff are most notable for what they do not contain…”, “This isn’t a very satisfying defense, for two reasons…”) are in reference to an initial email from Reinhart & Rogoff, not the actual response. Barro mentions that fuller response much later in the article “Early this morning, Reinhart and Rogoff put out a more detailed response saying…”
I hate to say it because you are normally so thorough, but in this instance your blog post is not up to your usual standards.
Having followed this issue elsewhere closely since the beginning, I must concur that this is not a post which would pass on the standards that you usually apply to yourself and others.
Linking to second hand commenters while ignoring what the researchers actually said is very disappointing.
So what you’re saying Johanna is that when the Marcott FAQ came out we should only have listened to RealClimate and ignored any criticisms?
records
The major difference is that when the Marcott FAQ came out, Steve actually read it before critiquing it. In this case he tried to draw analogies between R&R and his experience with climate science without – it seems – actually reading what R&R had already put out.
That doesn’t in any way come close to Steve’s normal standards.
Lest anyone think we only ever target climate science for pushing weak science into policymaking circles, Bruce McCullough and I published a
report in 2009 on the problem of basing policy decisions on studies that haven’t been independently checked, often due to non-disclosure of the underlying data. We mainly focused on the poor disclosure practices in economics but we also included examples from a variety of areas of science.
As Professor McKitrick implies, another area of similarity is what appears to be an extended delay before R&R’s “proprietary” data set was released to allow an independent examination to be performed.
Yes, Ross, an excellent paper. I’d lost the link and been unable to find it since. Many thanks for posting it.
Links #2 & #4 to Bloomberg [your “here here” etc.] are identical. Perhaps you might change one to this.
The original problems with McIntyre came from insisting things be provided as an Excel spreadsheet. He later denied it, but now he admits it again!
Steve: This was a total fabrication on Mann’s part. We produced emails demonstrating the fabrication in 2003 when Mann first made it. Mann repeated the lie to the Penn State Investigation which accepted his lie without checking.
I presume you’re referring to Mann’s oft-repeated claim that Steve asked for his data in an Excel spreadsheet because we were too stoopid to be able to use an ASCII file, and errors were introduced at that stage that we were too stoopid to notice. The truth of the matter, also oft-repeated but not enough, evidently, is that we never asked for an Excel spreadsheet and Rutherford only ever supplied the data in the .txt file format which was apparently the one he himself was using at the time.
When Mann first put out the claim about the excel file we released the email correspondence showing his statements were untrue.
Click to access Response.Oct29.pdf
Tim Osborne’s email from Oct 2003 shows that others saw through what Mann had said:
http://foia2011.org/index.php?id=5680
Mann has continued to peddle the spreadsheet myth, even to the Penn State inquiry, whose acceptance of it at face value without any fact-checking is an indicator of the overall quality of their investigation.
It was a joke, but yeah that’s what I was referencing.
The original problems with McIntyre came from insisting things be provided as an Excel spreadsheet.
The original e-mails on this issue are available, and you’re completely wrong. I hope you’ll look up the documents and admit that you’re peddling misinformation.
To MikeN
So you are one of those hit and run cowards who make an inflammatory statement that is proved patently false and we never see you in that thread again?
Yes (apparently)
My comment from yesterday about this on the previous thread, which takes a different perspective from our host, appears to be caught up in moderation. Is there some reason or a simple backlog?
There isn’t any pre-screening moderation here, you’re just getting caught in the spam filter for some reason.
I doubt it. I got “caught” by this mysterious “filter” only when I happen to write something Steve strongly disagreed with, even if it was expressed in a quite polite and modest way, such as my earlier critical remark of Ross McKitrick’s refusal to go to the Hartland conference. And my other comment on this thread is also apparently caught by this very smart “filter”, although it did not contain any offensive statements at all.
Wrong!
The filter will often catch you when there are too many external links (where “too many external links” is defined by the filter).
My comment was short and had no links. It still got caught in moderation.
steve, WordPress does strange things if you’re not logged in when you comment on another WP site, especially if you have more than one WordPress blog.
Re: stevepostrel (Apr 18 18:03),
The WordPress filter is more sophisticated than just links. It detects potential spammers via some methodology. (The fact that you have a web-link in your posting ID probably loses you points for a spam checker; link-spam is by far the most common form of spam…)
CA also has a method that attempts to catch political or religious or nasty speech. (Those are always-banned topics…)
No auto-system is perfect, and yes your surmise of a backlog is essentially correct 100% of the time here. 😦
I also had a comment stuck in the filter. It was neither critical nor contained a load of links. Its reproduced below. It would be good to know what it was which flagged it up for moderation.
Steve: in addition to multiple links, there are a number of words which trigger moderation. The word “Obama” for example triggers moderation and most such comments are deleted as political. There are many other blogs where people can express political views.
Reinhart and Rogoff had an error in THEIR excel spreadsheet. Steve has not asked for nor gotten their data and that is irrelevant here. What on earth are you referring to?
This was in reference to MikeN above
Can the R&R issues not be easily resolved by comparison with the Cook et al papers with regards to temperature reconstruction?
hehe. thought you’d like it
It might be simpler if you just wrote: “It was my idea to make the parallel with the hockey stick “.
As an economist the eerie resemblance between “climate science” and the “dismal science” as far as data handling is concerned, has always struck me.
The following abstract is from a very recent paper, and might be an eye-opener.
Abstract:
We provide evidence for the status quo in economics with respect to data sharing using a unique data set with 488 hand-collected observations randomly taken from researchers’ academic webpages. Out of the sample, 435 researchers (89.14%) neither have a data&code section nor indicate whether and where their data is available. We find that 8.81% of researchers share some of their data whereas only 2.05% fully share. We run an ordered probit regression to relate the decision of researchers to share to their observable characteristics. We find that three predictors are positive and significant across specifications: being full professor, working at a higher-ranked institution and personal attitudes towards sharing as indicated by sharing other material such as lecture slides.
Reference: Versbach, P.A. and Mueller-Langer F. (2013): Open Access to Data: An Ideal Professed bot Not PRactised, RatSWD Working Paper Series No. 215
http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2224146
Thank you for this. I’ll be reading this one tonight.
Economy and climate science: both tried to predict huge complex chaotic systems through computer models and failed. The former in 2008, the latter this decade. Opportunistic politicians and administrators rode both predicted “sure” waves delivered by certain scientists.
I think Steve is being patently unfair here. When requested, Reinhart and Rogoff provided the original working spreadsheets with all code. There was no stonewalling (though it would have better to make the code publicly available at the outset). In addition, when obvious errors were identified, they fully acknowledged their mistakes and thanked Herndon et. al. for identifying them.
Zeitlin’s comments which Steve quotes above is also clearly in error (“Yesterday’s defenses from Reinhart and Rogoff are most notable for what they do not contain: a defense of the methodology that Herndon and his colleagues criticize.”). On the contrary, Reinhart and Rogoff make a full throated defense of their weighting methodology. See http://www.businessinsider.com/reinhart-and-rogoff-admit-excel-blunder-2013-4
Making a comparison between Reinhart and Rogoff and Steve’s experiences in climate science is really strained.
“One response has been to argue that the causation is backwards, or that slower growth leads to higher debt-to-GDP ratios. Josh Bivens and John Irons made this case at the Economic Policy Institute. But this assumes that the data is correct. From the beginning there have been complaints that Reinhart and Rogoff weren’t releasing the data for their results (e.g. Dean Baker). I knew of several people trying to replicate the results who were bumping into walls left and right – it couldn’t be done.
In a new paper, “Does High Public Debt Consistently Stifle Economic Growth? A Critique of Reinhart and Rogoff,” Thomas Herndon, Michael Ash, and Robert Pollin of the University of Massachusetts, Amherst successfully replicate the results. After trying to replicate the Reinhart-Rogoff results and failing, they reached out to Reinhart and Rogoff and they were willing to share their data spreadhseet. This allowed Herndon et al. to see how how Reinhart and Rogoff’s data was constructed.
Read more: http://www.nextnewdeal.net/researchers-finally-replicated-reinhart-rogoff-and-there-are-serious-problems#ixzz2QqswKh2k”
#################
looks like they didnt make their data available before they did make it available
http://www.cepr.net/index.php/blogs/beat-the-press/not-following-professional-ethics-matters-also
I think this is unfair.
Reinhart & Rogoff replied within hours and redid the analysis overnight. They admitted what was unambiguously an error (but did not matter much) and corrected it.
The remaining issues matter more, but are opinion and judgement rather than true or false.
There are clearly points of difference in the RR situation. Readers, please don’t over-interpret my comparison of events to imply that everything matches. As matters unfold in RR, there appear to be many points of difference as well. Not least of which is that RR conceded some points. Another huge difference is that RR did not attack their critics on a personal level. Nor did they lie or fabricate about their correspondence with their critics.
There are, though, many problems with this entire blog post (see my earlier comment under Ivan Jankovic’s link to the WSJ piece).
– You extensively quote “Zeitlin” but don’t provide a link to the article that contains most of those excerpts.
– You attribute most of those excerpts to Zeitlin, when in fact they came from an unlinked article by Josh Barro.
– You don’t provide any link to, or analysis of, the actual response that R&R gave, nor do you quote what they said, even though their rebuttal was (from the time stamp on the WSJ response) published more than a day before your post.
– That means you ended up relying on other people to paraphrase what R&R had said and done, rather than looking at what they had actually said and done.
– The quotes that you used from the Barro article (“Yesterday’s defenses from Reinhart and Rogoff are most notable for what they do not contain…This isn’t a very satisfying defense…”) refer to the initial – almost instantaneous – email response, not the more detailed response that they put out a day later. Barro refers to (and links to) that fuller response towards the end of his article, but you chose to use his complaints over the initial email.
– It is totally premature to compare R&R’s response to climate-science behaviour just days into this, especially since you are not basing your commentary on R&R’s actual response.
None of this is to take any sides on the economics of R&R’s response to the critique, but it seems to me that R&R have done just about everything right in their response so far, from a procedural perspective. (I’m not trying to make any comment about earlier attempts to get data from them – I haven’t followed that side of this discussion.)
Meanwhile, the likelihood that you hadn’t seen the R&R response – as I expect you would have mentioned it if you had – means that you have been relying on very incomplete information and undermines the relevance of any comparisons you are trying to make.
However Gober, you can’t say that:
“It is sobering that such an error slipped into one of our papers despite our best efforts to be consistently careful,” they said in a joint statement. “We do not, however, believe this regrettable slip affects in any significant way the central message of the paper or that in our subsequent work.”
Doesn’t at least remind you in some non-zero way about what one theme of responses of Mann et al to MM criticism..?
http://news.yahoo.com/student-took-eminent-economists-debt-issue-won-095347790–business.html
You’re permitting yourself the freedom to accept a completely evolving ‘response’, at which point you cannot blame SM for discussing what was out there at the time by point out stuff that came after the fact.
Yes, it seems that R&R are being much more proactively capitulatory than Mann et al had/has been — the ideological difference is pretty obvious though: R&R’s paper goes against the ‘prevailing’ ‘fashionable’ economic philosophy (Krugman et al), and thus any faults are more readily leapt upon and nuances less tolerated. M&M were on the other side of the spectrum, themselves challenging what instantly became the prevailing understanding, and thus it is they with the burden, and Mann et al provided all manner of shelter. Recognizing the environment in which you operate/publish affords you a completely different posture when it comes to these situations.
Salamano,
No, I’m pointing out that the blog post had a number of clear errors, that Steve had ignored what had already been out there for over a day by the time he wrote it, and that he was instead relying on what other people said about R&R, rather than looking at what they said.
In isolation it might do, but they do go on to discuss the accusations in detail. Whether that detail is adequate or convincing is another matter – and I don’t think it is helpful to get into the economics here – but their reaction to the error is worlds apart from a Mann-style response.
As I said, by comparing the two situations, I am not taking the position that the details are precisely the same, only that some issues occur in both. It appears that RR were more forthright than the Team; I hope so. I am not in a position to debate RR issues. If I’ve missed some nuances, my apologies. I don’t have time right now to do a detailed fisking of the comparison.
I think you’re missing an important part of the story whereby each of the two situations finds the authors on the reverse sides of the tracks.
I’m assuming you are seeing this, but the reaction is huge on the part of Keynesians (prevailing economic understanding) leaping and piling on with regard to R&R’s paper, one whose conclusions they’ve always been predisposed to detest. Since they are also in the position of power and prevalence in today’s ecopolitical culture, this necessarily factors into the response pattern, no matter how much natural good-faith nature you impute to Reinhart-Rogoff.
Imagine instead if they had been on the opposite side of the coin, and their paper had been the seminal move in establishing the prevailing understanding of the state of their field, adhered to by those also in the positions of power within the field, only to be needled by scoffing outsiders– You definitely do get the sense that part of their initial response yearned to receive that same sort of circle-the-wagon benefit that Mann et al has enjoyed when responding to quite similar issues with their work.
R&R’s subsequent responses have an additional effect of manifesting their minority status within the field, even if it also appears to place them on a better trajectory than Mann et al with which they’ve been compared.
Reflecting on the enantiomeric images is illuminating.
===================
Academics and office nerds would be surprised how harsh and cruel blue-collar professionals unfairly slam the work of others. Coddling people who screw up and make excuses is enabling. Your profession might get more respect from the public if you took the work more seriously than your fragile egos.
Salamano, it’s not clear that Krugman’s position is as dominant. Yes he won a Nobel Memorial Prize, but so did Milton Friedman and von Hayek and Mundell.
Steve, Apr 19, 2013 at 10:49 AM | Permalink
The issue is that you made the comparison – you’ve written an entire blog post on the similarities. Talking of “fisking” the comparison gives the impression that fact-checking one’s own work is an optional hassle. The R&R response isn’t a “nuance” that it is acceptable to miss.
“The defences from Reinhart and Rogoff are also identical to Hockey Team defences” – among a number of similar comments – is hardly beating around the bush.
You’ve kept very high standards by being scrupulous about what you post. When you understandably objected to Karoly accusing you of “promulgating misinformation”, I argued on warmist pages that that, whenever you’d been inaccurate in anything you’d said or surmised, you’d corrected it as soon as you realized it – in clear contrast to the climate-science fraternity.
I’m not trying to defend R&R’s work – I’m astonished at the incompetence. But their response was to almost immediately admit the error and address the other criticisms head on. You don’t see that much in climate science.
I’m also not suggesting that R&R should even be a priority for you – you have much more pressing things to do. But if you don’t have time to get the basics right, there’s really not much point in writing the post.
There’s a slightly larger issue here, that gober and others are getting at, which I’ll try to give my take on, although it will take longer than I like.
Climatologists, whether of the warmist stripe, but also in the skeptical camp, which I am sort-of but not blindly partial to, seem to me not to live up to that neutrally objective ideal that we might all imagine ourselves aspiring to. Notwithstanding the laudable efforts of Steve here and others elsewhere.
Part of this less-than-ideal stance is brought upon us, I would guess, by the even more close minded and determinedly advocate position of the warmists and the damage they could be doing if, as seems likely, they are wrong in their prescriptions. But that shouldn’t really be a cause or excuse, and if we could get further toward the ideal, we’d probably have a better chance with the larger public. For just one example, let’s admit that the current 12-17 year pause in warming is not–yet–definitive proof of the warmists’ errors; such pauses in the long term record are really not abnormal in my view, and this current pause could reverse itself in a few years yet again…without in and of itself doing too much damage to theory on either side.
I’m following the particle physicist/cosmology debate about the origins of everything, and I see a much closer, although far from perfect, adherence to the ideal over there. Although personal ambition and competition takes place there too, there seems to me also to be an attitude, at least from time to time, of WANTING to be proved wrong, an eagerness to chase after apparent contradictions between theory and data, a glee that there is confusion on an issue and therefore some real science to be done. Wouldn’t it be great if climatology could be that way, and that there were no “sides” in this matter at all. Nature, after all, isn’t necessarily binary.
The physicists, in my view, are a bit closer to the ideal, the economists seem to be somewhere in the middle, and the climatologists–as a WHOLE– need to do some catching up. Least that is the way that I see it.
Meanwhile, I hope that some lurking climatologist has quietly taken up my suggestion that we use the techniques that the econ crowd used to analyze causality and reverse causality, and get us some comparable graphs for the CO2 and temperature confusion in the Vostok ice cores.
http://www.nextnewdeal.net/rortybomb/guest-post-reinhartrogoff-and-growth-time-debt
Steve has asked us not to discuss economic issues, but I’ll comment on some points made above. MikeN, re Salamano’s comment on “the ‘prevailing’ ‘fashionable’ economic philosophy (Krugman et al)”: Krugman’s Nobel for economics related to his early and excellent work on international trade theory, which I drew on in my policy work. He’s now widely regarded by economists as a partisan and far less rigorous commentator in line with what a Climate Etc poster calls “pissant progressives,” pushing policies which are popular in those circles rather than widely supported by economists.
I’m with Richard Tol on this one.
Looks like a case of getting to the correct answer in a somewhat unfortunate way.
They made their data available, acknowledged the error and responded without shooting/denigrating the “skeptics”.
We are a] a far cry from standard Team operating procedures and b] and R&R’s core argument is broadly and deeply borne out by empirical data.
The R&R controversy has been a bit over done. One thing I’ve respected about Steve is that he argues a study is wrongly done, but he doesn’t dismiss the entirety of global warming science. Instead his suggestion is that we realize comparing climate today to 1000 years ago is tricky business and the claims either way should be taken as weak given the methods now in use.
The “I told you so, austerity is bad” attitude has been pervasive in the leftwing blogs and this is not a conclusion that follows from finding this flaw in R&R which was never a study policy makers paid much heed to because of the lack of a theoretical explanation for why there should be a kink and discontinuity in the derivative.
Greg Manikaw had a very sober view on this which is more what I would have expected Steve to say. Instead I felt this post was a bit of pandering.
For the record, I agree with Richard Tol and nevilb. The comparison is unfair; if the climate people acted like RR we would be in a much better place today on the issues covered by this blog.
For the record, the Universe we happen to live in is unfair.
I have to concur. This post is atypical of most done here and is most unfortunate. Perhaps SM will assess himself a premature penalty for piling-on? 😉
It’s done in a mysterious way, but the ice here is always fresh enough for the skillful skaters.
============
And if not, the Zamboni is never far.
An overlooked climate-relevant aspect of the debate has been this analysis of reverse causality:
http://www.nextnewdeal.net/rortybomb/guest-post-reinhartrogoff-and-growth-time-debt
Has this technique been used on the ice cores that seem to show CO2 lagging temperature rise?
One difference between climate science and economic modeling is that in the physical world extremes are bounded. If you plot the heights of humans on earth you may get a few approaching 8 ft tall, but never anyone 10 ft tall. In economics, you can have enormous extremes, millions and millions of folks who make about $30-50k per year, and then a few billionaires. Economics has what N. Taleb calls “fat tails”, significant numbers of outliers, and boy are they significant. In the real, physical world, when you see a 6 sigma outlier, you suspect something funny. In climate, that could be very rapid tree growth due to recent damage, could be silt run-off after a big storm, etc.
(By the way, Taleb in his new book, “Antifragility”, has some really blistering comments for economic models and modelers, and medical researchers too.) In economics, those fat tails can be really good, such as mega book sales for a runaway bestseller or
hit movie, or really bad, such as in stock market crashes and banking collapses.
A more fundamental fault than what is being discussed here is zeitlin seems to have got the GDP for New Zealand, at least according to this Reserve Bank data
Click to access dp09_12.pdf
It makes one wonder where the data sets came from.
In 1951, as well as the surge then slump in wool prices from the Korean war, there was a five month watersiders strike that severely affected shipping. I don’t know how that affected matters, especially with regard balance of payments. New Zealand also has an April to March financial year which will cause differences to calendar year data, especially if there are sawtooth effects.
A shrewd observer commented at Naked Capitalism yesterday: “Reinhardt/Rogoff: I had reservations about their thesis, in being sceptical about there being a magical threshold point (at whatever %age might be claimed), and about trying to identify causation from correlations involving multivariable systems. …
I’m not particularly sceptical about generalised warnings against accruing piles of debt, mind, but about the notion that obsessive number-juggling concerning ill-measured quantities is likely to be a source of scientific enlightenment.
Come to think of it, the thrust of these remarks covers “climate science” pretty well too.”
“As each successive and desperate Marcott or Gergis paper comes out, it’s just a matter of spotting which they’ve mugged; the data, the methods or both.”
With papers like RR, you have to take a view as to whether you’re dealing with basic incompetance or a willingness to “bend” the numbers for a higher cause.
Pointman
Odd that “unfisked” is part of your lexicon.
Richard S.J. Tol
Posted Apr 18, 2013 at 4:12 PM
Thanks, Richard. I have mixed feelings about this, in part because there are mixed errors. Let me start by saying I take no position on their intentions in all of this. It may indeed be coincidental that all three of their errors were in the same direction. Odds on that are one in four, so you can’t reject the null hypothesis.
The underlying error was that they did not make their spreadsheet publicly available when they published the original study. Their results were quite sobering. It showed a cliff with things getting bad quickly past 90% debt/gdp. Here’s my chart showing their results, which were sometimes called “the cliff” and were used around the planet to justify a variety of conclusions and actions.
The problem, as with the hockeystick, is partly in the overselling of the conclusions. However, from their results it was clear that big government debt was lethal to the economy, so it was a big conclusion … one that folks couldn’t replicate for some time.
Now, three years later, and with much ink spilled, here’s what the updated and at least more correct figures show:
What is humorous in this have been their attempts to avoid saying “We were wrong”. Oh, they admitted the Excel mistake in fine and noble fashion, but regarding the rest the best they can say is:
Well, yes, their estimate for the median is the right order of magnitude … so what? The “cliff” in their results, the cliff that occasioned so much remark and supported so many arguments, is an illusion. It never was.
So Richard, the funny part and the part that is parallel to Mann’s hockeystick is watching the gyrations these guys go through to try to resurrect a totally dead study. The claim that brought both Mann and these authors fame was an error, a mistake, a wrong reading of the data, a mathematical mess-up.
That must be hard for someone to deal with … and I have a small inkling of that from bitter experience, I’ve been very publicly wrong in some of my scientific studies. Fortunately, I wasn’t famous for any of the wrong ones.
Best thing to do is to admit it, and recant the erroneous results. Don’t try to soften it al all. Don’t claim some other part of the paper somehow rescues the “cliff” as they are doing. They need to say something like “We were wrong, and we advise everyone who depended on the idea that debt over 90% was lethal that we no longer hold that to be even approximately true.”
Instead, they’re backing and filling about the freakin’ median being of the “right order of magnitude” like that makes any difference.
Mann has devoted his entire career to trying to prop up the “hockeystick”. Why? It’s what he’s famous for, it’s the only thing he’s ever done, it’s his sole claim to fame. He can’t admit it was an illusion. And near as I can tell, these poor buggers are in the same boat, trying to prop up the “cliff”. Like Michael Mann, the only reason anyone ever heard of Reinhart and Rogoff is for an erroneous, incorrect claim …
Anyhow, that’s why it’s funny, Richard, at least to me, hope that helps although explaining a joke often doesn’t work all that well.
My best to you,
w.
Willis’s sociology is wrong in that Rogoff was already a mainstream big shot prior to the work with Reinhart. (Also a chess grandmaster prior to going into economics.)
He once got into a public tussle with Joseph Stiglitz over Stiglitz’s criticisms of IMF stabilization policies (he used to work at the IMF); he’s been a respected, more-or-less establishment, policy-oriented international macroeconomist for many years. Not at all like Mann’s position when he published the hockey stick.
Thanks, Steve. I didn’t mean he wasn’t a big fish in the small pond of economic analysis. He was, as you point out, well known in that world.
I meant what I said, that like Michael Mann he became famous because of one and only one thing … and like with Mann, that thing turned out to be in error.
By famous I mean discussed around the world and his (erroneous) results receiving global circulation and promotion. Famous like when he goes down it gets discussed here, and is widely reported in the mainscream media.
Regards.
w.
Here is another analog between Climate Science and R-R. For years, R-R has been paraphrased in the financial industry roughly as follows: recessions caused by financial crises take longer to heal than other crises. Therefore, the current policies of borrowing, spending and printing cannot be held accountable for the continued sluggish economy. Now, R-R may have been rebutted and the paraphrased meaning of the rebuttal is that borrowing, printing and spending do not damage the economy. Whether it’s right or wrong, R-R bolsters the status quo.
It appears the issues with R&R can maybe be resolved with climate science. There are certain aspects of economics, no matter how much they think is true based on economic laws, that simply do not manifest in the climate at large. By making observations about the climate and the underlying economic factors, we can make judgments about whether those factors in fact influence the economy in the way the economic modelers claim. Perhaps this can be explained by DC at a non-agressive economics blog over the course of a month.
I would agree with some of the posters here that there are differences and similarities between the R_R incident and what we see in climate science publications. The similarity is that papers from individuals, who on obtaining results that agree with a foregone conclusion, I think tend not to look for errors in their works as diligently as they would have had the data pointed in the other direction. The difference is when a paper goes against the consensus it tends to be thoroughly reviewed by others in the field as contrasted by the fact that papers supporting the consensus are not so scrutinized.
Finding individual errors in a paper instead of looking at what could or could not be a compendium of errors sometimes leads to the situation where the author can deal with the individual error and come away by declaring it did not matter when in fact it is combined errors that can invalidate a conclusion. Of course, the other way off the hook is to point to a minor error or perceived error in a critique and claim that that alone negates the entire critique.
I will not put a particular economic theory forth here but certainly when studying the effect of the ratio of government debt/GDP on GDP growth one must somehow avoid the circular effect that GPD includes government spending that in turn these days can effect government debt or as we have seen recently in the US the only method of increasing spending is through debt.
Lately there have been some heart wrenching tales about deleted comments, and some may be surprised to learn that my own genius has been rejected from time to time, so take heart friends; there is still a Pulitzer category and jury for Drama.
You might find the discussion ongoing at the political science blog, The Monkey Cage, about the data methodology and replicability of the R&R study http://themonkeycage.org/2013/04/19/what-the-reinhart-rogoff-debacle-really-shows-verifying-empirical-results-needs-to-be-routine/#comments
Willis, I’m skeptical of any analysis of a very complex subject in which there is much nonsense peddled – such as economics. Like climate science, many researchers overlook basic data – the popular notion that low interest rates and low unemployment cannot co-exist, for example is disproven by simmply looking at actual US data.
Like climate science, results vary with the researchers underlying view of humans – creative vs not, untrustworthy vs acting for life.
Like climate science, there are objective people in the field. Northrop Buechner, John Ridpath, and Richard Salsman come to mind.
I agree that R&R are in a tough spot, of their own making – taking the defensive as people often do. It’s a matter of pseudo-self-esteem. (At least one group put their climate alarmist paper on hold or such.)
(Two problems with stonewalling are:
– eventually the truth may out
– when it does, the defensive person’s reputation is sunk)
And like climate science, WordPress software is not highest quality work. I guess it’s what is affordable in the absence of “big oil” money. 😉
PS: Yes, much sloppiness about – for example Mann’s error pointed out by M&M in vol 16 No 1 of Energy and the Environment. Their claim is that Mann’s actual methodology did not match what he stated, but he claimed that had no effect. (One of many cases in which alarmist analyses are incorrectly sensitive to a subset of the data, and denial of the errors.)
Can I just make one point everyone here, including, Steve seems to have missed. When you get a degree in economics you get an “arts” degree, when you get a degree in climate science you get a “science” degree.
However much economists like to add mathematics to try to make it a science we all know it remains an arts subject. The R-R paper, ignoring the calculation errors, like many economic papers raised interesting political and policy questions. However, no one in economics would ever claim any idea was proved nor that any model is infallible.
What intrigues me is that the BBC have covered this but have generally never given air time to the Teams misdemeanour and Steve’s and Ross’s work on the statitistics used by Climate Scientists?
http://nymag.com/daily/intelligencer/2013/04/grad-student-who-shook-global-austerity-movement.html
3 Trackbacks
[…] https://climateaudit.org/2013/04/18/the-hockey-team-and-reinhart-rogoff/#more-17835 […]
[…] The Hockey Team and Reinhart-Rogoff (climateaudit.org) […]
[…] was wrong, and the authors have admitted their mistake. (Bizarrely I discovered this fact at Climate Audit, a blog devoted to the statistics of climate […]