While I’m often described as a “statistician”, as that’s a word that people understand (or think that they understand), I think of what I do more as “data analysis”. Academic statisticians are interested in different sorts of things than interest me. I have some styles, habits and practices for approaching new data sets, but they probably derive more from osmosis from geologists looking for anomalies on geophysical maps than anything that you’d learn in a statistics course (Hans Erren understands this exactly).
I ran across two articles by Jan de Leeuw (1998, 1994), a prominent applied statistician in the social sciences, towards the end of his career, philosophizing about what applied statisticians actually did or (could do) and was struck by the aptness of many of his observations for present-day climate science.
de Leeuw has a nice turn of phrase, so the articles are fairly lively. The abstract to de Leeuw 1994 read
When, after the agreeable fatigues of solicitation, Mrs Millamant set out a long bill of conditions subject to which she might by degrees dwindle into a wife, Mirabell offered in return the condition that he might not thereby be beyond measure enlarged into a husband. With age and experience in research come the twin dangers of dwindling into a philosopher of science while being enlarged into a dotard.
As someone who’s past his best-before date, it was impossible for me to resist reading the article. Here’s another zinging epithet from de Leeuw:
Science is presumably cumulative. This means that we all stand, to use Newton’s beautiful phrase, “on the shoulders of giants”. It also means that we… stand on top of a lot of miscellaneous stuff put together by thousands of midgets.
Another nice epithet:
It is a truism that statistics cannot establish causality of relationship. It is quite incredible, by the way, that most people who quote this result are engaged on the very same page in trying to accomplish what they have just declared to be impossible.
de Leeuw’s reflections tie statistics to “data analysis” and de Leeuw 1988 ends:
Statistics is data analysis. This does not mean that we want to replace the academic discipline “statistics” by the academic discipline “data analysis”, it merely means that statistics has always been data analysis.
Substantively, de Leeuw takes special note of the following form of data analysis:
We usually do not want a small and uninteresting perturbation of our data to have a large effect on the results of our technique.
and
Classical statistics has always studied stability by using standard errors or confidence intervals. Gifi thinks this is much too narrow and other forms of stability are important as well.
This latter point is highly on point for the sort of analysis that we’ve explored here. This (of course) is a staple CA methodology, though, of course, I look for such perturbations quite differently than the Team. My typical sort of observation is that a typical Team reconstruction is not “robust” to bristlecones, Yamal vs the Polar Urals Update, etc. These issues seem so humdrum that it’s hard to imagine that intelligent people cannot instantly grasp the point. However, in reply, we see longwinded expositions of that their results are “robust” to something that’s always somewhat different than the issue in question – that they are “robust” to whether MBH proxies are weighted uniformly or with MBH98 weights (who cares?). Or that if you have both bristlecones and Gaspe, they are “robust” to take one away sensitivity. Or that they are robust to Tiljander (if bristlecones are in) or robust to dendro (if upside-down Tiljander are in.)
I’ve paid negligible attention here to the calculation of “standard errors or confidence intervals” other than to deconstruct and demystify wildly over-confident Team claims. Some CA readers (though not me) routinely demand error bars for calculations where no one really knows how to calculate them. If you don’t know how to calculate error bars, what’s the point? (The flip side is – if you only know how to calculate “minimum” error bar (and the actual error bar is much greater, should you be graphically presenting this “minimum error bar” given the risks that readers may interpret this representation as being a usual error estimate .)
I won’t bother summarizing the articles because they are easy reads (See links in refs below), but here are a few quotes from de Leeuw that caught my eye.
It is not true that people first formulate a model, then collect data, and then perform statistics… The model gets adapted in the process, various modifications are tried and rejected, new parameters are introduced, and so on… the decisions made by the scientist cannot be formalized before the data are collected…
as we have seen many times in the non-scientific world, rules and laws that continue to exist although nobody obeys them and takes them seriously merely lead to hypocrisy…
Another problem, which is related to the first one, is that the models typically proposed by statistics are not very realistic. Especially in multivariate situations, and especially in the social and behavioral sciences, the assumptions typically made in the standard statistical models do not make much sense. Data are usually not even approximately normally distributed, replications are often not independent, regressions are not linear, items are not Rasch, and so on…
the prescriptions of classical statistics easily leads to hypocrisy. The confidence intervals and tests of hypotheses of statistics are valid only if the model is true. Because we know that the model is never true, not even for an idealized population, it is not clear what we must do with this statistical information. This does not mean, by the way, that the models and corresponding techniques are useless. On the contrary, most of the established statistical techniques are also very useful data analysis techniques. Otherwise they would not have survived. We merely must interpret our use of them in a different way than we are used to…
Statistical statements are not about the data that we have observed, but they are about a hypothetical series of replications under exactly identical conditions. It seems to me that such statements are not interesting for many social and behavioral science situations, because the idea of independent replications is irrelevant. Different individuals or societies or historical periods are not replications from some sampling universe, they are essentially unique. There is no need to generalize to a hypothetical population. All we can require in situations like these is an appropriate description or summarization of the data which illustrates the points the scientist wants to make and which documents the choices that have been made…
There used to be a time when statisticians and their cronies, the methodologists, always complained that they were consulted too late. Scientists only arrived at their offices after the data had been collected, i.e. after the damage was done. The implication was that a much better study would have resulted if the statistician had been consulted earlier…
The statistical priesthood on the other hand has two counter arguments. The first one is that you must assume something, otherwise you can do nothing. The appropriate answer here is that this is nonsense. I can compute a mean value and I can draw a straight line through a cloud of points without assuming anything…
Originally, of course, statistics was descriptive… the emphasis on inference and cookbook forms of instant rationalism becomes prominent with the Neyman-Pearson, decision theoretical, and Bayesian schools…
In order to prevent possible misunderstandings, we emphasize that the information that z = 1.96 is a useful descriptive statement, often more useful than the statement that the difference in means is 4.89…
In fact it seems to be the case that the social sciences clearly illustrate that there is nothing inherently cumulative and self-correcting about the development of any one particular science…
Our conclusions so far, on the basis of the above, can be summarized quite briefly. The task of statistics is to describe the results of empirical investigations and experiments in such a way that the investigator can more easily make his predictions and generalizations. Thus it is not the task of statistics to make generalizations. Statistical inference, whatever it is, is not useful for empirical science. Many statistical procedures are very useful, many statistical measures of fit provide convenient scales of comparison, and many statistical models provide interesting theoretical illustrations and gauges with which we can compare our actual data. But generalization, prediction, and control are outside of statistics, and inside the various sciences. Statistics has given us many useful tools and scales, but it has not given us a methodology to make the appropriate inductions…
Suppose we compare correlation matrices in terms of some numerical criterion, which can be the determinant, the eigenvalues, multiple or canonical correlations, or whatever…
Basing techniques on statistical models is an extremely useful heuristic device. Many other useful heuristic devices exist, for example those based on graphs and pictures. The statistical methodology “behind the techniques” that is usually taught to harmless and unsuspecting scientists is a confusing and quite nonsensical collection of rituals. Many of the techniques work, quite beautifully, but this is despite of and certainly independent of this peculiar philosophy of statistics…
This is supposedly “sticking out one’s neck”, which is presumably the macho Popper thing to do. There are various things problematic with the prescription. They are by now tedious to repeat, but here we go anyway. In the first place, if you follow the prescription and your data are any good, your head gets chopped off. In the second place, because people know their head will get chopped off, nobody follows the prescription. They collect data, modify their model, look again, stick their neck out a tiny bit, modify their model again and finally walk around with a proud look on their face and a non-rejected model in their hands, pretending to have followed the Popperian prescription.
PS: I confess that I “adapted” the first epithet a little. de Leeuw’s own phrase had, perhaps, a different nuance:
Science is presumably cumulative. This means that we all stand, to use Newton’s beautiful phrase, “on the shoulders of giants”. It also means, fortunately, that we stand on top of a lot of miscellaneous stuff put together by thousands of midgets.
References:
De Leeuw, J. 1994. Statistics and the sciences. UCLA Statistics Preprints: 1–20. http://repositories.cdlib.org/cgi/viewcontent.cgi?article=1058&context=uclastat
de Leeuw, J. 1988. Models and techniques. Statistica Neerlandica 42: 91-98. http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.49.3338
Climate Audit 
85 Comments
Re: statistician vs. analyst
A statistician is the guy who innovates methods. An analyst is the guy who applies them. I think serious problems occur when you get analysts thinking they’re innovators. Which is exactly what has happened in climate science. There’s an inherent conflict of interest when the guys who invent the tests and the guys who apply them are one and the same. Sort of like the guys who build GCMs being the same as those who search for AGW fingerprints.
Re: bender (#1),
I’m not sure that I make the distinction in precisely that way, though we undoubtedly agree on any salient point.
Let’s take something humdrum like Briffa et al Phil Trans B 2008. In effect, they argue that some weird tweak on multivariate methodology is the “right” way to analyse dendro data and from this, they can show that something or other is unprecedented.
If their interest is promoting the method, then you’d think that they would concurrently publish their R or Matlab code and deposit the package in the software archives so that others could apply this new mousetrap to other data sets and vindicate the method – as opposed to making other people run the gauntlet of trying to figure out what they did.
Re: Steve McIntyre (#2),
In the field of cryptography, if someone designs a new cryptographic algorithm and tries to sell it, most cryptanalysts are particularly dubious of its quality if the details of how it works aren’t shared. If the algorithm isn’t open source it becomes somewhat harder to duplicate but it probably provides less security than an open source algorithm. The idea is called “security through obscurity” by the cryptographic community.
From the preface of Bruce Schneier’s Applied Cryptography:
“If I take a letter, lock it in a safe, hide the safe somewhere in New York, then tell you to read the letter, that’s not security. That’s obscurity. On the other hand, if I take a letter and lock it in a safe, and then give you the safe along with the design specifications of the safe and a hundred identical safes with their combinations so that you and the world’s best safecrackers can study the locking mechanism — and you still can’t open the safe and read the letter — that’s security.”
I think the same concept applies here. The security, in this case, is in how they feel about their position, but that’s not what they’re really practicing by keeping their data and methods a secret. There’s absolutely no reason for it.
Fortunately, as in cryptanalysis, weak methods can be exposed through diligence, and the original algorithm is not required in order to break it. Many times, more elegant mathematical solutions to a cryptographic problem are devised simply because the original isn’t available.
I would have thought the problem even worse, that our giants are now standing inan ever expandinga sewer lagoon being excavated with the latest in automated equpment.
This is one of the most important points to me. So often, when you hear a scientist called out with “correlation is not causation,” they respond “well, of course, but….” Everyone claims to accept the rule, but each individual thinks that his/her particular situation is different, and deserves that exceptional “but.” The will to believe is just too powerful to overcome strict adherence to what they know is right.
I am good friends with the Chair person of the Biology department at a major University here in the US. We were discussing Statistics and causality and he maintained that “Of course linear regression proves causality. After all, he continued, one of the variables is called the explanatory variable and the other is the response variable and they wouldn’t be called that if the first didn’t cause the second.”
Scary. This guy is bright, hard working and well published. Scary.
Re: John Knapp (#5),
There is a major difference between an ad hoc correlation in a bunch of survey data and regression analysis of a carefully controlled, randomized, replicated experiment. Your friend is right.
Re: bender (#23),
Sorry, Bender, but both you and John Knapp’s friend are entirely wrong. We need to get that word “prove” out of scientific discussion. Negative results can “prove” that a given hypothesis (or even a theory) is incorrect, but it can never “prove” that it IS correct. It can be cited as “evidence of”, and “indication of”… but NEVER “proof of”. Facts (and data) never “speak for themselves” but are always filtered through a theory of some kind. The same facts can often be used to support two radically different theories.
Re: bender (#23),
Perhaps a bit of middle ground on this. If I were to have a reason to belive that A would effect B, such as increasing CO2 levels would increase plant growth, I could design an experiment to test this. As part of the design I would hypothesize that if my theory were true I would expect to get a positive correlation when I regressed the data. Then if after I did the experiment I got that positive correlation I could say that the results gave evidence that my hypothesis was true.
This is a very different statement than saying that a positive correlation between CO2 levels and plant size proves that CO2 is a fertilizer.
In the first the causation is postulated based on the scientist previous knowledge, expertise in the field, previous experimental results, etc.
As De Leeuw says:
Thus it is not the regression that goes to causation. It is the scientific theory that is supported by the existence of a positive correlation that does.
Steve: OK, enough on this issue.
I always thought data analysts were the people who look for the information which the data might contain, while statisticians devise ever more complex ways to show how data might contain any amount of information, pointing in any desired direction.
That is why politicians employ statisticians, while commercial organisations employ data analysts….
– Newton
– Abelson
You said, “PS: I confess that I “adapted” the first epithet a little”
I defintly prefere the one with the word “ fortunately “, not because it make us look bigger, but because it induce that midgets, too, built the world.
Sorry of dwindling into a philosopher of blog 😉
Science is a cumulative process, to be sure, but there must also be a filtering process before, during and after cumulation in order to catch mistakes and self-deception, – snip – and to properly assess the real value of the accumulation to society as a whole.
What I worry about is not the acculumation, but the filtering process -snip- which has caused some work to be assimilated into the body of science which shouldn’t be there and blocked accumulation of vital information which should be there.
Steve: I take it that you commented before reading the articles or else you would have noticed an apt comment by de Leeuw that I chose not to excerpt to see if anyone would notice. There are others:
De Leeuw seems pretty harsh on Statisticians, “statisticians and their cronies”, “The statistical priesthood”. Why is that?
Re: Jeff Alberts (#10),
perhaps because he is one. However, I think I understand the point that he’s making and it’s worthwhile in the present environment where people tell climate scientists to get thee to a statistician. All too often (and this is on the authority of von Storch who’s as thoughtful about this sort of thing as anyone), academic statisticians aren’t much help with practical problems.
de Leeuw’s articles are a reflection on why that is.
If someone asked me for a statistical reference summarizing the methods used here, I really don’t have any suggestions.
Re: Steve McIntyre (#11),
I guess I can understand that. I supposed the self-deprecation is similar to tactics used by prosecuting attorneys, get the dirty laundry out up front so the defense can’t make a big issue of it.
Re: Jeff Alberts (#12), I don’t think that that’s a fair description at all. I don’t think that he was trying to get things out of the way – not that there’s anything wrong with such a practice (I wish climate scientists would do it, so you can isolate issues.) I thought that de Leeuw was making honest reflections on an interesting problem.
Re: Steve McIntyre (#13),
I agree there, I wasn’t really speaking to the point, but his use of certain phrases describing those who practice Statistics.
I don’t agree with the conclusion. Apart from correlation you need more proof to establish causality. If you have the extra proof, you can establish causality, if you don’t, you can’t. The way I understand the quote, It seems he doesn’t acknowledge the possibility of adding more proof to establish causality.[ Steve: did you read the article? That’s the opposite of what he says.]
Here is a question to ponder about; Does lack of correlation proof lack of causality?
And what is the statistical validity of a measurement if you don’t know how to calculate its error bars? Or maybe I should rephrase the question. Can you trust a measurement if you don’t know how to calculate its error bars? [Steve: again, I urge you to read the articles before commenting off the cuff on what you presume it says.]
Another way to summarize all this — a lot of people would benefit from a course in the Greek tragedies in order to appreciate the ever present danger of hubris. There is far too much certainty and far too little appreciation for the possibility that assumptions are wrong (in their own work and the work of others). A mature self-awareness and appreciation for human imperfection seems to be in short supply.
Re: stan (#16),
Spot on Stan,
“A mature self-awareness and appreciation for human imperfection seems to be in short supply.”
One thing I’ve learned from my own personal experience in life is to be very weary of any one who claims to be an expert on a given subject and who makes confident assertions about something on the basis of ‘argumentum ad verecundium’ (argument from authority).
In the final analysis IMO any statement made by anyone, whether they be an expert or not, is just their opinion. It is up to them to justify their statement and just because they claim to be an expert means diddly squat IMO. Sadly it seems to be very rare to see any kind of humility amongst so called (all too often) self declared experts these days.
KevinUK
As my philosphy professor told us in PH101, 30 years ago, the red traffic light does not cause the green traffic light (or vice versa), even though they are 100% correlated.
Re: DaleC (#17), I think the traffic lights are not a very good example. They are correlated, and there is an underlying model behind their behavior (the fact that it is a design does not matter). So if one isolates a signal between green light and red light without any a priori knowledge, one can not rule out that there is something intersting to study.
Thank you for the quotes. de Leeuw writes very beautifully, full of confidence yet wonderfully circumspect.
Alex
Prof Tipler wrote:
“Increasingly, government grants are used to defend dogma, not discover new truth: 28 percent of the scientists supported by NIH admitted recently to cooking data to support establishment theory, and 66 percent admitted to cutting corners to achieve the same end.
I myself no longer trust the data claims appearing in the leading science journals.”
Anyone know where he got these numbers?
Regarding error bars:
My understanding is that their relevance is of the order of:
If everything I have stated is true and the model I have presented is valid then …
As it is almost inevitable that something I have stated is not true and in some aspects the model is invalid then they are to be largely ignored.
Perhaps they are the measure of the strength of an argument be it true (or false?) but they do not it seems to me imply much more.
I accept fully that data can and should have error bars if they are subject to quantifable errors, but that is data, I am puzzled as to how they apply to arguments.
Ho Hum
Alex
Ward,
I am no cryptographer but I worked on the testing of chipped credit cards, this is a very open source area.
I get a a little crest fallen by the papers that rely heavily on statistics as I suspect them. I cannot decode them. Fortunately others can but only after great effort. To me they are encrypted but fortunately to others (many of them here) they are merely obscured.
Alex
Re:Bender #23
The “regression analysis of a carefully controlled, randomized, replicated experiment” is a descriptive statistic of the results of the experiment. The experiment and its design may go to show causation but the regression, if for no other reason than the variables can be switched and the same level of correlation will result, doesn’t speak to the causitive factor at all.
Curieux (#8), as for your preferences, you certainly would not have missed the ample ambiguity and delicate irony behind that confessed ommision, if you had considered what de Leeuw wrote on his estimate of what midgets tend to do to social science every once in a while.
A delightful exposition on the dangers of simplification of reality, when trying to find truths hidden by the complexities of existence. Producing numbers is not the same thing as properly planned statistical analysis.
A great deal indeed, thanks for the link, interesting reading and for me personally its that long bill of conditions that keeps tripping me up…
The quotes from De Leeuw are indeed delightful. I’d come up with this word independently (i.e. before reading #29), and it still strikes me as being appropriate. (Here I have to confess that I have not yet read the full piece). I also felt pleased that the iterative cycle proposed by De Leeuw closely resembles what I’d written as an internal report in about 1975, which I called “The Aim of Experiments”, in an effort to induce researchers to think in terms of a plausible underlying model, carry out their procedures and fit their data to the model. The next step is to re-think the situation depending on the outcome of the the preliminary fitting process and a possible change in the “Aim” of the work. The question should be “Can I improve the direction of my efforts as a result of considering the outcome of prior work?” Based on my experience experimenters do this all the time, I think, but often only on a rather restricted scale. A semi-formal extension of the principle to the complete experimental field might stand many scientists in good stead.
In regard to the quote from Newton
I read a biography of Newton many years ago. It asserted that Newton made this assertion ironically. He was referring to Leibnitz as the giant and the dispute over the development of the calculus.
Re: TAG (#33), I believe it was widely understood at the time to be a reference to Hookes, who was extremely short in stature.
Re: ad (#35), Oops, Robert Hooke.
Re: ad (#35),
That’s also the story I read, and further, it was a cruel, sarcastic dig at Hooke, who was hunchbacked.
(and now I’m off to read the article)
Re: PaddikJ (#61), PadickJ:
In case you missed my response elsewhere – many thanks for recommending Tey’s The Daughter of Time.
Re: ad (#34),
Ironic. If Hooke obeyed the law that gave him fame, then he would grow taller when those above him got off his shoulders.
Good header, Steve, highly entertaining and so relevant.
Re: Geoff Sherrington (#36), Newton was almost certainly just reciting a standard tag line in Western philosophy. See http://en.wikipedia.org/wiki/Standing_on_the_shoulders_of_giants.
Re: dixon fiske (#59),
I don’t rely on Wikipedia. The material you reference contains “attributed to” type terminology and seems drift away from the philosophy of science to life in general. My comment was a light-hearted response to the application of “Hookes Law” to a spring under a lesser load.
Re: dixon fiske (#59),
From Wikipedia:
Newton was master of the Mint in his later years and the Royal Mint has fallen for Newton’s joke
“..If I take a letter, lock it in a safe, hide the safe somewhere in New York, then tell you to read the letter, that’s not security. That’s obscurity….”
Bruce Schneier
“He that would keep a secret must first keep it secret that he hath a secret to keep…”
Lucius Annaeus Seneca, famously quoted by Sir Humphery Appleby
Again, the comparison between science and politics is instructive. In science if you try to hide something it will be noticed when it eventually comes out. In politics, if you can hide something until the next election is passed, the critical vote is taken, or the publishing deadline is reached, you have won, and no one thereafter will care about the issue. You will have ‘moved on’….
Correction is SteveM’s burden. Also:
Eisenhowers Farewell.
Those interested in data analysis might find these pieces interesting:
http://projecteuclid.org/DPubS?service=UI&version=1.0&verb=Display&handle=euclid.ss/1076102418
http://projecteuclid.org/DPubS?service=UI&version=1.0&verb=Display&handle=euclid.aoms/1177704711
A year ago it became clear to me that I needed to access Science again, to try to help restore what looked like a complete derailment of Climate Science. I was constantly constrained to ask fundamental questions “what is Science? what is Scientific Method?” and “what do we need to develop and strengthen in Climate Science to restore a working balance?” and big, simple principles became clearer. Like the importance of transparency for replication. Yet in conventional Climate Science, this is not even missed. The whole science abdicates to pseudo-statisticians, computer games players, and politicians, who all claim to have the grasp of the whole subject that other lowly specialists no longer dare challenge.
After nearly dying through NOT trusting my intuition, I’ve learned the high importance of taking exact, careful note of such faculties. Imagination, that is normally considered a non-scientific faculty, is actually essential for perception – for noticing subtle and significant patterns. It also gives us the ability to cope through the dry patches that have led so many to lose belief in their ability to see the whole picture, and specialize, abdicating to the alarmists.
It is important that statistics, sometimes regarded as the most nefarious tool of manipulators, can here, through transparency, be applied reflectively to itself, potentially to help heal the damaged science. It can be doctor as well as pirate. But this needs the imagination, that I felt Steve was subtly pointing towards with phrases like “other forms of stability” [than standard errors and confidence intervals].
PS Brunel had a similar kind of near-death experience at the beginning of his career, I believe.
Journal of the Royal Statistical Society: Series D, vol. 47, part 2 (1998), has a section devoted to discussing how much of statistical research is just seemingly-impressive mathematics that has little or no utility in real data analysis. Here is quote from the paper by D.J. Hand (p.247).
I have found it difficult to discuss things like this, because not everyone seems able to distinguish between disputes over statistical assumptions (which can be legitimate) and disputes over statistical calculations (which cannot).
For myself, I would like to see more Monte Carlo simulations.
The bloke who taught me introductory stats said that the elementary methods could be justified by a list of assumptions, but that much of their usefulness, in his opinion, came from their familiarity and objectivity. So if someone said “y was fitted as a function of x using unweighted linear least-squares”, anyone would get the same result from the same data. All that’s required is universal agreement on the algorithm, and the complete reporting of the data. It seems to me that Climate Science often fails by his criteria. The data is not transparently reported and the algorithms used are unfamiliar and underspecified. It’s a blue do.
Today’s scientists have substituted mathematics for experiments, and they wander off through equation after equation, and eventually build a structure which has no relation to reality.
– Nikola Tesla
Jan de Leeuw can certainly turn a pretty phrase or two and does it well in telling his story. The message I receive is that nothing is simple in how statistics is applied and that it is important that users/consumers know the limitations of statistics.
I do not see him making any profound judgments about statisticians or data analysts. In my view, the man could understandably be a bit cynical working as a statistician in the social sciences, but I do not see that either. He did imply that the social sciences do not seem to make good use of the additive processes in establishing theories. I think to some degree the same could be said about economics – or at least in the applications and justifications for those applications.
I would like to hear what the statisticians, who post here regularly, think about this piece.
What are “the agreeable fatigues of solicitation”? Is it because stats is usually a service industry, that the Popperian ideal is not realized. Stats is reduced to finding ways to confirm the clients result, and even more so if the same person is doing both the stats and the research. Not wanting to go off into off-limits motivational, but all this discussion about the way things ‘should’ be done, needs a dose of reality. As a friend of mine at a biological research school used to say — we are all biostitutes.
Re: David Stockwell (#47),
I think that, sadly, too many of those who are advocates of science and many scientists themselves reject the conclusions Popper came to. Whilst scientific conclusions can be put to great utility, I don’t believe we should ever stop questioning those conclusions. The falsifiability tenet was hard enough to win, but many wish to give no further ground than that. Too many, I fear, believe that science can arrive at hard conclusions which cannot be falsified. That attitude is to the detriment of the practice of science, in my opinion.
Too many comments on this thread are piling on. de Leeuw had some sensible points which I had hoped that people might comment on, instead of focusing on the piling on issues.
I’ve never met a playful anti-authoritarian that I didn’t like. Until now. That first paper is awfully pessimistic and unhelpful. What a curmudgeon.
.
This paragraph has applicability everywhere. In medical device manufacturing, the FDA is currently on a statistics kick. Everything must be justified statistically. This drives people to rote use of statistics without understanding what it is the statistics actually tells them. As long as a confidence interval can be put on the process, the process is good.
.
This leads to all kinds of problems. The design may be able to tolerate dimension X +/- Y, but because someone did a Ppk analysis that yields X +/- Z, parts get inspected to +/-Z. If Z is smaller than Y, you end up with unnecessary waste. If Z is larger than Y, you end up with unnecessary rework.
.
Statistics can also lead you to faulty conclusions – especially with the more complex reliability metrics, like sequential life testing. A lot of assumptions go into those statistics, assumptions that are not necessarily fulfilled by the test being conducted. But it looks good on paper, no?
.
To me, the importance of the paper is in trying to demonstrate that statistics is a tool – but only one of many tools – that can be used for scientific investigation. This need to term everything statistically in order to have a conclusion perceived as “legitimate” results in the over-use and mis-use of statistics. The RE/CE statistics used by Steig, et.al., to show concordance between their differing reconstructions and actual surface records is a great example of this. That tool is not useful for determining the accuracy of trends of data subsets, and it’s easy to show why not. Developing confidence intervals using imputed data points artificially inflates the degrees of freedom and results in intervals that are way too small.
.
Over-reliance on statistics results in a lack of critical thought.
Re: Ryan O (#51),
.
Those are meant to be 2 wholly separate examples . . . which wasn’t entirely clear when I re-read after posting.
Re: Ryan O (#50),
Ryan, those statements sum it up nicely for me. I would add that one needs a statistician to inform the user of statistics the limitations of said statistics. I also would hope, that the author’s critical remarks about the misuse of statistics, is not taken as an indication of the uselessness of statistics. Statistics is a critical tool in all areas of science and thus one that needs to be applied correctly.
Also I like your use of a period between paragraphs to better set apart those paragraphs. These old eyes thank you.
bender,
I do a lot of analyses that fall into what de Leeuw would call “stability analysis” – you know the sort of thing – what if you use Indigirka River or Polar Urals Update instead of Yamal. Or 2 covariance PCs instead of 4 covariance PCs. Or Ababneh instead of Graybill. Things that surely fall into the category of “small and uninteresting variations” of data selection that have disproportionate impact on results.
We are also familiar with Team responses – that they’ve done some other sort of analysis, which proves that their reconstruction is “robust” and that whatever variation I’ve studied is somehow “wrong” and not just “wrong”, but perhaps the stupidest variation in the entire history of science.
It’s helpful to me to have a text that places value on the idea that “small and uninteresting variations” should not affect results.
“playful anti-authoritarian” – hmmmm, wonder if there are any of those around here? 🙂
Re: Steve McIntyre (#52),
Would stability analysis be the same as sensitivity testing? Those analysis by whichever name could be critical in determining whether the authors of a study overlooked something or (and better left unsaid) the data handling was being gamed to present the conclusions in a better light.
Re: Kenneth Fritsch (#54), seems like the same thing to me.
Re: Steve McIntyre (#55),
It is the same thing. Sensitivity testing is critical, often overlooked, and the sort of thing SI appendices were designed for. Reviewers who forget (conveniently or otherwise) to ask for sensitivity testing are IMO categorically “negligent”.
Re: Steve McIntyre (#55),
This is the kind of analysis where you hide away the adverse results that demonstrate the lack of robustness of your analysis to the inclusion or not of certain HS shaped producing proxies in a folder called ‘censored’ isn’t it Steve? Maybe it should therefore be called ‘censorship’ rather than ‘stability’ or ‘sensitivity’ analysis?
KevinUK
well, deLeeuw didn’t actually write the turn of phrase you so enjoyed SM.
When, after the agreeable fatigues of solicitation, Mrs Millamant set out a long bill of conditions subject to which she might by degrees dwindle into a wife, Mirabell offered in return the condition that he might not thereby be beyond measure enlarged into a husband. With age and experience in research come the twin dangers of dwindling into a philosopher of science while being enlarged into a dotard.
That is attributed to: C. Truesdell
Himself an interesting case.
Enough pedantry.
Steve: I presumed that it was a quote from something – it was just such an odd quote for an abstract that it was irresistible.
Steve Mc
quote:
“I have some styles, habits and practices for approaching new data sets, but they probably derive more from osmosis from geologists looking for anomalies on geophysical maps than anything that you’d learn in a statistics course”
I think that’s one of the nicest things I’ve heard said about my profession in over 35 years 🙂 BTW, I’m sure you have other epithets for us, and all …
Re: ianl (#62),
I learned that sometimes you have to read between the lines.
Re: ianl (#62),
Geologists look for geological features.
Geophysicists look for geophysical anomalies.
Geochemists look for geochemical anomalies.
The three sub-disciplines require rather different approaches and measures, but certainly success is enhanced by their successful combination by managers who, usually expert in just one of the above, can appreciate the advocacy and distribute an optimal funding balance.
Four major differences with climate science as so far publicised are
(a) if we do not succeed, we go out of business
(b) we do not succeed consistently, if at all, with dubious data
(c) we look for the explanation of anomalies; we do not try to smooth them out of significance, or discard them lightly
(d) when we succeed, we create new, tangible wealth and materials for the benefit of all people.
Re: Geoff Sherrington (#66),
a little too much editorializing.
Re: Steve McIntyre (#67),
OK.
Whether it be this or that, the data calls you in some wierd way. It intriques you beyond belief.
I liken it to a musical, a more mathematical drive towards a human existence tuning. But, there is something that isn’t in tune in many of these articals you review. Does being out of tune mean nothing anymore.
If we listen to a different group, accentuating strings and brass instead of guitars and drums, it sounds different.
It is OK to consult an quantum mechanic or two about the claim of a radiative 1.2 C for a doubling of CO2, no? Can we go here?
Re: EJ (#64),
Warning: Steve M prefers to focus on mainstream literature, especially that cited by IPCC. THAT’s where he wants contributors to go.
Re: EJ (#64),
Steve knows a certain squash playing ‘wannabe Green Party councillor and paleo-climatologist’ who is actually an ‘atmospheric scientist’ who doesn’t bother to discuss quantum mechanics with his colleagues who work in the offices just down the corridor from him. Anyone like to guess who this might be?
Clue
KevinUK
Re: KevinUK (#72), Connolley?
Re: KevinUK (#72),
No – our friend Juckes! (Yes, I needed the clue.)
Re: bender (#74),
Sorry bender looks like I got my info slightly mixed up (early onset of alzheimers due of course to global warming). You are correct Billy ‘the wiki’ Connolley is definitely a squash player but I’m not so sure about Juckesy as there are two squash playing Martin Juckes that I’ve found courtesy of Google. While rechecking I came across this interesting link
M N Juckes personal BADC page
Who would have guessed that Juckesy is a member of Greenpeace?
Also found this entry in Bill’s diary back in 2000
Bill knows Jucksey and Peter ‘Day of the Triffid’ Cox
KevinUK
Steve: Looks like Martin Juckes of the Royal Navy here is the Martin Juckes playing for the Defence Academy in the Oxfordshire league here. He seemed a little too sour to be a squash player. Connolley is not as sour.
Re: KevinUK (#75),
I’m intrigued. What do you mean by ‘too sour to be a squash player’. Do you mean that he’s a poor loser? By chance I was just preparing a light hearted reply to your #63 comment above about “I learned that sometimes you have to read between the lines.” (I presume you are talking about absorption lines?) when I came across this useful link about water vapour absorption lines.
PhysicsWorld article
The article documents the many problems that exist in attempting to understand the ‘real’ greenhouse gas i.e. dihydrogen monoxide. I think the last paragraph is particularly important and should be borne in mind by all climate modellers.
“It is clear that the absorption of radiation by water vapour determines many characteristics of our atmosphere. While we would not try to provoke any worldwide movement that was aimed at suppressing water emissions, it would seem that the climatic role of water does not receive the general attention it deserves.”
KevinUK
Re: KevinUK (#75),
So which one is the guy that won’t talk to his colleagues down the hall?
Re: bender (#77),
The ‘too sour to be a squash player’ one to quote Steve.
KevinUK
Very interesting discussion, I agree with Geoff haha Cheers!
Meanwhile, Down Under, the national newspaper, “The Australian”, informs us on 7 March 2009 as follows:
Sceptics are mad as well as bad, John Naish writes in The Ecologist
Press release from the University of the West of England:
In my mental instability I have forgotten when the next full moon shall appear and cause me to make wolf noises.
Re: Geoff Sherrington (#79), Needless to say, of course, being in a panic about CO2 is the only rational state.
Let’s lay out the bare bones here:
1) GCMs are unreliable and cannot parse the distribution of energy from increased CO2 forcing through the climate.
2) The IPCC itself actively misrepresents certainties and ignorance.
3) Paleo-temperature reconstruction is a circus of almost clownish negligence.
4) The surface air temperature record rests on poorly maintained weather stations.
1-4 have been demonstrated. And so,
5) There is zero scientific evidence that increased CO2 has influenced Earth climate at all.
And so now we have activists, eco-psychologists and psycho-sociologists presuming to lecture gravely on the pathological mental state of “deniers” who, by all scientific accounts, are actually those adhering most closely to the knowledge of the matter. I wonder how many of them have read the primary literature; or even read past the executive summary of the 4AR SPM.
The real lesson here is the loss of rational sight to the blinding light of pious certainty. One day, maybe, some psycho-sociologist will find a career path in describing the fatuous nonsense that resulted from the psycho-grip of eco-religion.
Re: Pat Frank (#81),
The consequences are being felt already. The birth rate will drop. Who feels romantic after reading papers about the psycho-grip of eco-religion that you mention?
re: above – thanks Geoff full press release here:
http://info.uwe.ac.uk/news/UWENews/article.asp?item=1438
Does anyone have a suggestion for a single link which Mr Marshall should be pointed towards that may give him another perspective? The Wegman Report? Other?
Well, that was nice to read. Those papers are 20 years old, interesting they still have some life in them.
Re: Jan de Leeuw (#83),
We live in interesting times, sir. Old writings take on new meaning in exceptional circumstances.
Re Jan de Leeuw, #83,
Welcome to CA, Dr. de Leeuw! Unfortunately, Steve McIntyre is on the road for the next 2 weeks or so, but I am sure other regular readers will have comments for you.