Here are some thoughts from 70 years ago from the eminent statistician, R.A. Fisher, from the “Design of Experiments”. There’s a nice dig about “heavyweight authorities”.
When any scientific conclusion is supposed to be proved on experimental evidence, critics who still refuse to accept the conclusion are accustomed to take one of two lines of attack. They may claim that the interpretation of the experiment is faulty, that the results reported are not in fact those which should have been expected had the conclusion drawn been justified, or that they might equally well have arisen had the conclusion drawn been false.
Such criticisms of interpretation are usually treated as falling within the domain of statistics. They are often made by professed statisticians against the work of others whom they regard as ignorant of or incompetent in statistical technique; and, since he interpretation of any considerable body of data is likely to involve computations, it is natural enough that questions involving the logical implications of the results of the arithmetical processes employed, should be relegated to the statistician. At least I make no complaint of this convention. The statistician cannot evade the responsibility for understanding the processes that he applies or recommends.
My immediate point is that the questions involved can be dissociated from all that is strictly technical in the statistician’s craft, and when so detached are questions only of the right use of human reasoning powers, with which all intelligent people, who hope to be intelligible, are equally concerned, and on which the statistician as such speaks with no special authority. The statistician cannot excuse himself when the duty of getting his head clear on the principles of scientific inference, but equally no other thinking man can avoid a like obligation.
The other type of criticism to which experimental results are exposed is that the experiment itself was ill designed or, of course badly executed. If we suppose that the experimenter did what he intended to do, both of these points come down to the question of the design or the logical structure of the experiment. This type of criticism is usually made by what I might call a heavyweight authority. Prolonged experience, or at least the long possession of a scientific reputation, is almost a prerequisite for developing successfully this line of attack. Technical details are seldom in evidence. The authoritative assertion “His controls are totally inadequate” must have temporarily discredited many a promising line of work; and such an authoritarian method of judgement must surely continue, human nature being what it is , so long as theoretical notions of the principle of experimental design are lacking — notions just as clear and explicit as we apply to technical details….
In the foregoing paragraphs, the subject matter of this book has been regarded from the point of view of an experimenter who wishes to carry out his work competently and having done so wishes to safeguard his results, so far as they are validly established, from ignorant criticism by different sorts of superior people.
I hope that I’m fitting into the first category; you can probably guess my views on the rest of the story.