As some readers have observed, there is a lively controversy regarding an influential recent paper by Reinhart and Rogoff. Herndon et al (of Raymond Bradley’s UMass-Amhertst) concluded that RR’s conclusions depended on a bad weighting method, inexplicable exclusion of data from certain countries and years and even an Excel coding error. All the sorts of issues that are familiar with Mann and the Hockey Team. Even the defences from Reinhart and Rogoff are eerily similar: no errors ever seem to “matter” because of some new and unfisked study. One big difference: Reinhart and Rogoff at least conceded things that were unarguable, whereas Mann and the Hockey Team concede nothing, not even things as incontrovertible as upside down use of contaminated data.
Without dwelling on the underlying economic issue, the parallels to the Hockey Team are unmistakable. A critique stated that their conclusions depended on three errors:
a bad weighting method, inexplicable exclusion of data from certain countries and years, and an Excel coding error.
Precisely the sort of problem that has been discussed at Climate Audit.
The defences from Reinhart and Rogoff are also identical to Hockey Team defences. Zeitlin states:
Yesterday’s defenses from Reinhart and Rogoff are most notable for what they do not contain: a defense of the methodology that Herndon and his colleagues criticize.
Instead, like the Team, in response to criticism of any given article, they point to some other article – generally one just hot off the press which hasn’t been fisked yet.
The business writers are less forgiving than environmental writers. Zeitlin observes:
This isn’t a very satisfying defense, for two reasons.
One is the simple sloppiness of the apparent error that Reinhart and Rogoff made in calculating the average GDP growth for high-debt countries. Barry Ritholtz put it well: “Whenever you hear the phrase ‘the fundamental case is intact,’ you best run screaming from the room.” It’s reminiscent of the “fake but accurate” defense of the forged memos about President George W. Bush’s service in the Texas Air National Guard.
The Reinhart-Rogoff error also seems to have depended on small datasets with puzzling exclusions. Zeitlin:
Dean Baker has a good breakdown of what went wrong. What happened is that Reinhart and Rogoff included data from New Zealand in 1951, a year when debt/GDP exceeded 90 percent and GDP growth was -7.6 percent. But they excluded New Zealand data for the period 1946 to 1949, when debt also exceeded 90 percent and growth was often very strong. Include those years, and New Zealand’s average economic growth in high debt years is not -7.6 percent but +2.6 percent. Because there were only seven countries in the data set that Reinhart and Rogoff used to calculate average GDP growth under high debt conditions, and because they weighted each country’s average growth equally, getting New Zealand wrong by more than 10 percentage points was a very big deal, shaving 1.5 percentage points off their estimate of average growth.
The style of analysis in Reinhart and Rogoff is also very similar to the style of analysis in paleoclimate: relatively small data sets. In both cases, the analysis is statistical, but not exotic statistics.
It turns out that much of the resonance of the original Reinhart-Rogoff work depended on a compelling graphic: their “cliff chart”. Zeitlin:
Unlike the cliff chart that made the Reinhart-Rogoff paper so arresting, this way of looking at the data suggests that the historical performance of countries with varying debt-to-GDP ratios has little to tell us about what today’s fiscal policies should be.
Critics also observe that the authors have been highly involved in advocacy:
Reinhart and especially Rogoff have spent years now engaged in a high-profile political advocacy campaign grounded in a causal interpretation of their empirical work that both of them knew perfectly well was not in fact supported by their analysis
Maybe Herndon and coauthors will now take in an interest in work by their UMass colleague, Raymond Bradley.