or “principal mode” or something like that.

Interestingly, the concept of “modes” is rather common in the signal processing community. At least, the SIGINT guy I used to work with (before I got this shiny new job) referred to eigenvalue decompositions as dominant mode analysis. I was explaining the Gram-Schmidt process I was working with (uh, a sort of PCA method, btw) and he kept mentioning that I was “finding the dominant modes” (he also kept bringing up “Householder rotations”). It was a term I was unfamiliar with, but nevertheless meant exactly what I was addressing.

Mark

Thinking some more about it, I see one huge difference – the weather forecasting models are physics-based, even if there is a lot of parameterization in them and they actually do get tested on out-of-sample data. So one can see how the term “skill” can have real meaning in a practical sense. When one is making a climate reconstruction from dendro site chronologies, we do not have a physics-based link between ring widths and temperature – we have curve fitting in a calibration period. I think that the enterprise is much more in the style of statistical modeling than it is to weather forecasting.

You have summarized my thoughts on the subject nearly exactly. Without out-of-sample results be careful of what and how you conclude, be very careful. Heck, you should even be careful of how you conclude using out-of-sample results.

And a statistic used more or less exclusively in your community or invented in your community should not be considered merely a convenient term for community discussions, it should either have statistical rigor or be redefined as a part of your community terminology and without statistical implications.

Outsiders with sufficient talents and hard work can do things that the insiders can’t. They can be patent clerks or mining consultants and their way of talking about scientific questions doesn’t have to fit the box of a horde mentality, but this itself doesn’t prove that they’re wrong.

I like that .
Actually I even think that the more complex a field is the more chance an outsider has to shed some light in it because he is still able to ask simple , relevant questions .
I have a friend who is expert in the string theory and he told me that the guy who will find one day the way out of this mess will almost certainly be somebody who has not a clue in string theory and who will ask the right question about the gravity and quantum mechanics .
Somebody I forgot has said that it is better to bring wrong answers to relevant questions than to bring right answers to irrelevant questions .

I don’t think that the observations “network of statisticians hasn’t heard about XY” proves that XY is wrong.

I think that, for the most part, most of us are on the same page here, although we may be expressing it in different ways. My instinct when I see discussions of “statistical skill” is to see if that is isomorphic to some statistical concept with which I (or Wegman) might be familiar – in the same way that I don’t throw up my hands because someone calls what I would call an “eigenvector” an “empirical orthogonal function” or “principal mode” or something like that.

Von Storch has observed to me quite fairly that most statisticians don’t really bring much to the party as they typically arrive with i.i.d. baggage and try to force the problems into a mold that doesn’t apply. I thnk that econometricians and even business statisticians are, in some ways, more useful because they are used to autocorrelated series and have different instincts than an i.i.d. statistician.

When one is dealing with the Team, I get the sense of people using big words pompously or people trying to fit the facts to the policy without inquiring into what could be wrong with their model rather than practical people soberly viewing their data.

Let’s suppose that we grant weather forecasters their use of “skill” as a useful concept in eveluating weather forecasting systems. The question then is whether this vocabulary with its metaphorical baggage is a more useful way of approaching paleoclimate reconstructions. Thinking some more about it, I see one huge difference – the weather forecasting models are physics-based, even if there is a lot of parameterization in them and they actually do get tested on out-of-sample data. So one can see how the term “skill” can have real meaning in a practical sense. When one is making a climate reconstruction from dendro site chronologies, we do not have a physics-based link between ring widths and temperature – we have curve fitting in a calibration period. I think that the enterprise is much more in the style of statistical modeling than it is to weather forecasting.

Thinking about it a little more,

Another word to describe this approach is bias it and can’t be used consistently with the laws of logic. If a person writes a non-rigorous paper asserting “A”, someone else can write a non-rigorous paper asserting “non A”. You can’t assume that both of these papers are wrong just because you should always assume that non-rigorous papers are wrong – because it would be logically inconsistent.

I think that the only scientific approach is to admit “I don’t know” at the beginning and judge various statements fairly regardless whether they are formulated as “positive” statements or “negative” statements – which is a matter of presentation not science, after all. Saying that one must always assume something about a certain result formulated in a certain way is bad. Assuming that a paper must be wrong is as irrational as assuming that a paper must be right.

Of course that until sufficient evidence is given for a statement or a paper, a rational person won’t take it seriously. A rational person will continue to think that “we don’t know”. But saying that it must be wrong just because it appears in a paper that is not rigorous is simply irrational. Even non-rigorous work can lead to a systematic improvement of our knowledge, and in most fields it does as long as the signal is stronger than the noise where noise includes not only uncontrollable fluctuations but also fraud and ideologically-driven selection bias.

And surely the experimental particle physicists have written up and justified any new techniques they might have developed?

I just think you wouldn’t be satisfied. The colliders, for example, deal with a huge number of events (colissions that produce new particles with particular properties) per second and it is impossible to store data about each of them. Most of them must be thrown away. The events that are used to deduce physics are only an “interesting subset”. The discarded events are “uninteresting” and they are counted as following a boring background. Quite clearly, there is a lot of conventions one must pick in dividing the events into the interesting and uninteresting ones in the first place, and understanding physics is a key to do it right.

After you do it, you end up with the interesting events but there is no 100% reliable way to reconstruct what happened in a particular event. (In fact, this question can’t be answered even in principle because amplitudes from all possible processes with the same initial and final state contribute to a given event.) Many different types of events – e.g. with new intermediate particles – have signals that overlap with each other but give different patterns of signals, hopefully. Some of the signals are known, some of them are not because they are “new physics”. One must be able to choose a cut that defines the number of jets in an event – a jet is a stream of many particles going in a “similar” (…) direction, resulting from a quark or gluon that was in the middle.

There is no canonical method to do these things. In fact, there are many systems of conventions and approaches. Also, there are many event simulators, and they differ in profound technical details. You can’t justify one over the other by a universal argument. You must allow them to co-exist and only if one of them turns out to give systematically more useful and accurate results, it will be preferred. Everything must be allowed to compete: not only experimental teams and theories but also statistical strategies. Which of the event simulator strategies will be the winner in the future doesn’t depend on statistics only. It depends on the laws of physics, and some of them are not yet known or not yet properly understood as far as their consequences go.

I am sure you must have heard this from many alarmists already, but I certainly agree with whoever has said that science is not just statistics, and the best choice of statistical methods to study Nature may depend on the laws of physics, not just some universal statistical rules and dogmas.

It is also untrue that statisticians are necessarily guaranteed to be the best people for everything that has the word “statistics” in it, just like climatologists are not necessarily the best choice to solve anything with the word “climate” in it. The expectation whether someone does things right and takes the most important things into account depends on the intelligence and training of the person, and similarity to other tasks he has solved in the past.

I am really sorry to say the same thing as Mann but it’s true. Now, Mann was almost certainly wrong in most of his statements about the past hockey climate, which makes the work unusable, but you can’t prove he was wrong just because he is not a statistician. The whole concept of competing professions who have the right to keep ownership over their topics is completely silly. If someone else does things better than a “clique” in a field, he is just doing it better.

If string theorists become better in predicting nuclear scattering experiments than the people who learned nuclear physics for decades, what can you do about it? It is pretty likely that it is going to happen. There are many examples like that. Outsiders with sufficient talents and hard work can do things that the insiders can’t. They can be patent clerks or mining consultants and their way of talking about scientific questions doesn’t have to fit the box of a horde mentality, but this itself doesn’t prove that they’re wrong.

Dear Willis #34, I completely agree that in order for a method to have any value, one must show that it works. I don’t know why you think I disagree. But before you prove that it works, you must try whether it works.

Bender #35, this is just terminological issue but you’re just wrong if you say that heuristics can be rigorous. Internally they can be rigorous but they’re certainly not rigorous as a description what they want to describe, by definition. Heuristics are tools to direct our attention in the hopefully right direction. And again, I don’t think that the observations “network of statisticians hasn’t heard about XY” proves that XY is wrong.

the rather rigorous definition of SS in your comment combined with the opinion that it is just a “heuristic” just don’t seem to fit together too well.

Heuristics can be complex and they can be “rigorous” in their definition. That doesn’t mean a network of statisticians is likely to have heard of them. I’m not dismissing SS, or any heuristic. I’m explaining how gaps in knowledge between disciplines can arise. Creativity in isolation is how it happens. And this is the mark of art, not science.

I would discourage everyone from criticizing others for using certain methods just because you’re not used to these methods unless you are able to prove that they lead to wrong conclusions or results.

This is Alice in Wonderland science.

Alice laughed, “There’s no use trying,” she said, “one can’t believe impossible things.”

“I daresay you haven’t had much practice,” said the Queen. “When I was your age, I always did it for half-an-hour a day. Why, sometimes I’ve believed as many as six impossible [statistical procedures] before breakfast.”

The onus is not on us to believe six new statistical methods before breakfast, or to show that some novel, unusual statistical procedure doesn’t work. The onus is on the dendrochrologists to show that it does work, that is some theoretical justification for the procedure, and their conclusions should be discarded until such substantiation is forthcoming.

w.