High priests of data analysis

The rules that apply to ordinary followers of a religion do not necessarily apply to high priests. Ordinary Catholic men must remove their hats when they enter a church. This "universal" rule does not apply to the Pope and the cardinals.

Suppose that a company wants to introduce a new drug. Before the drug is approved, the authorities, doctors and patients expect the company to perform clinical trials, preferably with a control group, perhaps using the "double blind" protocol. The collected data have to be analyzed using the best available statistical techniques and only then the decision to approve the drug can be made. If the company proposed to approve a drug on the basis of general or philosophical arguments, without any supporting data, they would be ridiculed. Or they might be considered to be a representative of "alternative medicine", where scientific standards are largely ignored. In any case, a proposal to approve a drug without supporting data does not fit in any way in modern science.

The above simple and intuitive rules of scientific approach to medicine apparently do not apply to statistics. Statisticians are high priests of data analysis. They analyze data for all other scientists and thus make modern science possible. However, statistics is an exceptional science where methods can be approved and used without any evidence based on hard data. General or philosophical arguments are sufficient to justify a method. My sarcastic remarks are inspired by the article [1] The Significance Test Controversy and the Bayesian Alternative by Bruno Lecoutre and Jacques Poitevineau published in StatProb, an online encyclopedia. The article discusses various problems with significance tests and considers several alternative approaches to data analysis. The truly amazing aspect of this article is that the authors do not think that it is necessary to support their claims with data. If, as the authors claim, significance tests are imperfect and confidence intervals or "prep" give better results then there surely must be some empirical evidence to support these claims?

In this context, empirical evidence could include a count of correct or incorrect decisions, records of small or large discrepancies between estimated and true values, financial gains or losses incurred by users of statistics, etc. In some sciences data are very hard to collect because of technological limitations, difficult access, financial costs, etc. (consider, for example, collecting data on supernova explosions or human genome). Nevertheless, scientists strive to collect at least some data and refrain from far reaching conclusions until at least some empirical evidence is available.

The striking feature of the statistical debate exemplified in the article quoted above is that not only empirical evidence is not presented at all but there is no explanation for why it is missing. The unavoidable impression is that the evidence is missing not because it is hard to collect but because statisticians do not consider empirical evidence to be desirable.

I cannot read the minds of statisticians but let me offer a guess of why statisticians do not collect data about the performance of statistical methods. Statisticians collect or help to collect a lot of data. Data analysis is the essence and the bread and butter of their profession. They subconsciously think that this activity fulfills the scientific requirement for collecting data in support of statistical methods. Statisticians do not realize that collecting data to analyze various problems in other sciences is not the same as collecting data to verify the value of statistical methods.

I quoted [1] to give a concrete example of an article with a significant methodological gap. This article is by no means exceptional.