Mann connoisseurs eagerly await any offerings from the Maestro and, just as the Christmas season typically brings a new potboiler from John Grisham, so Christmas 2007 has brought us a new offering from the Maestro, this time on hurricanes. Readers will not disappointed.
The season brings us two closely related papers: Sabbatelli and Mann (JGR 2007) and Mann and Sabbatelli (GRL, 2007). The first paper purports to establish a relationship between Atlantic cyclones and independent variables: Main Development Region SST, Nino 3.4 SST and NAO. The second paper applies the relationship from the first place to estimate cyclone “undercounts” in the pre-reconnaissance period, settling on a modest undercount estimate of 1.2 storms/year.
Mann and Sabbatelli 2007 refers to an SI (doi:10.1029/2007GL03178) but there is no such SI as of today at ftp://ftp.agu.org/apend/gl/2007GL031781. Data versions are available at http://www.meteo.psu.edu/~mann/TC_GRL07/ without commentary. Separate files for each component of the regression can be identified.
The file TCcounts.dat has four columns with the year in column 1. Column 2 matches Hurdat basin counts for 1870-2006; column 3 adds an undercount of 1.2 for 1943 and while column 4 adds an undercount of 3.
The nino.dat file also has 4 nino variations that are highly correlated. I could not locate an explanation of the difference between the different versions. It is said to be:
The Nino3.4 index was taken from the Kaplan et al.  data set and updated with subsequent values available through NCEP. The
However, I was unable to identify in matching version in a couple of potential Nino 3.4 data sets listed below and temporarily gave up in frustration. (AGU has a policy requiring authors to identify replicable url’s, but the policy is not applied to the Team.)
I had more luck identifying a provenance for the NAO data. Sabbatelli and Mann stated:
The boreal winter (DJFM) NAO index was taken from Jones et al. , updated with more recent values from the University of East Anglia/CRU.
There is a data set at NOAA http://www.cdc.noaa.gov/Pressure/Timeseries/Data/nao.dat which exactly matches the CRU version http://www.cru.uea.ac.uk/ftpdata/nao.dat during their common coverage. Updated values are listed at http://www.cru.uea.ac.uk/~timo/projpages/nao_update.htm. Sabatelli and Mann stated that the winter indices are assigned to the December year.
the 1997/1998 El Nino and winter 1997/1998 NAO value were assigned the year 1997
I cross-checked this manually and confirmed that, for example, the 1997 value in Mann’s nao.dat (0.8000) matched the 1997-98 CRU value (+0.80).
I first made a data frame in which all series were assigned to the same year as the Mann data i.e. the row with the year 1997 contained the NAO for winter 1997-1998 etc. Sabbatelli and Mann says:
For simplicity, the “year” was defined to apply to the preceding storm season for both indices (e.g., the 1997/1998 El Nino and winter 1997/1998 NAO value were assigned the year 1997).
You have to watch a bit carefully here, because it turns out that Mann regresses the storm count against the following Nino and NAO indices not the predecessor. Mann justifies this with the tricky phrase:
However, we do find a statistically significant lagged correlation relating the Nino3.4 index to the MDR SST series for the following year’s storm season, consistent with the observation elsewhere [Trenberth and Shea, 2006] that ENSO events influence tropical Atlantic SST in the following summer.
If you’re not watching carefully, you’d assume that preceding winter’s climate data would be used to predict tropical cyclone counts rather than the following winter’s data. But you’d be wrong, as I’ll show shortly. Now for the purposes of estimating past tropical cyclone levels, as Mann and Sabattli try to do, this is not necessarily the end of the world. If there’s a relationship between 2006 cyclone counts and 2006-2007 Nino 3.4 and 2006-2007 NAO, one can utilize this for past estimates. But the causality relationship is certainly not what one expects and this surely warrants a little discussion.
I did both ordinary linear regressions and poisson regressions. The relative performance in different cases was virtually identical and I found it a little bit more convenient to do simple linear regressions as the diagnostics in R are a bit more elaborate. I got an r2 of 0.4647 using this model, which corresponds to Mann statement:
The statistical model captures a substantial fraction R2 = 50% (i.e., half) of the total annual variance in TC counts
I thought that it was very nice of Mann to provide an interpretation to climate scientists of the difficult % statistic. I’m sure that the explanation that 50% was in fact the same thing as one-half will be welcomed by his readers.
If you do the same calculation for the lagged NAO and Nino indices (or for the lead indices by one more year), the r2 declines to the 0.32-0.33 range in both cases. So the reported relationship is definitely with the following winter’s Nino and NAO indices. The assignment of the 1997-1998 Nino to 1997 was not just for “simplicity”, but to improve the stats. It’s hard to imagine that they didn’t also do calculations using the preceding season indices, discarding these calculations when they didn’t work as well. This sort of data snooping needs to be reflected in confidence estimates, but isn’t done here.
Early. Late and Middle Models
Mann and Sabbatelli 2007 report on two alternative calibrations of the model used for verification: one on the period 1870-1938 and the other for the period 1939-2006. They report that the r2 declined only slightly to 43%. (Speaking of which, didn’t Mann say somewhere that calculating r2 statistics would be a “foolish and incorrect thing to do”. Such merriment from the prankster.)
In the R-implementation of Poisson regression, the r2 statistic is not reported (and I didn’t, at this time, bother calculating it.) Since the linear regression moved in parallel with the poisson regression, I compared information using linear regression – recognizing that a more precise match would calculate the r2 using poisson regression. I got similar results, only a very slight decrease in r2.
Now here’s the interesting bit. Just for fun, I re-calibrated the model using a calibration period 1946-1992 and then 1930-1992. The r2 declined to about 0.2. So the model is not nearly as stable as advertised.
Following Mann’s method with 1930-1992 calibration (although using linear rather than Poisson regression – I could retool this with Poisson regression, but it won’t make anything other than a microscopic difference and it will take extra time not required for the point), I then compared the estimated count to the actual count below. In the calibration period, you can see a plausible fit. However after the calibration period in 1993-2006, you see that there are more storms than predicted using the “middle” calibration and prior to the calibration period, amore substantial; undercount. Given the low r2, I wouldn’t put a whole lot of weight on these results, but they are definitely inconsistent with the advertised results in Mann and Sabbatelli.
In any event, the performance of their model degrades substantially when the middle portion is used for calibration. Did they bother doing this elementary check and failed to report the results? Or did they omit this obvious check? Neither is a very good answer. This particular check was not very hard to do. I’ve probably spent about 5 hours on this study, much of which was consumed with attempts to verify data given the inadequate data provenance. If a reviewer was spending a few hours on the review of a paper like this, you’d think that they’d try an analysis like this and have got similar results.
So when you’re appraising exactly what you’re getting from a review, you need to bear in mind that the reviewer has merely expressed his opinions on the paper based on a read of the paper and that even minimal due diligence has probably not been carried out.
Mann, Sabatelli et al 2007. http://www.meteo.psu.edu/~mann/shared/articles/MSN-GRL07.pdf
Sabbatelli, T.A., Mann, M.E., The Influence of Climate State Variables on Atlantic Tropical Cyclone Occurrence Rates, J. Geophys. Res., 112, D17114, doi: 10.1029/2007JD008385, 2007.