We’re Using a Common Statistical Test All Wrong. Statisticians Want to Fix That.

P-values, or probability values, are  “commonly used to test (and dismiss) a ‘null hypothesis’, which generally states that there is no difference between two groups, or that there is no correlation between a pair of characteristics. The smaller the P value, the less likely an observed set of values would occur by chance — assuming that the null hypothesis is true. A P value of 0.05 or less is generally taken to mean that a finding is statistically significant and warrants publication.” (1)

Moved by growing concerns about the reproducibility of research, the American Statistical Association (ASA) issued a statement in March 2016 to address the widespread misuse of p-values. ¹

In this article, Retraction Watch supplies a set of six principles pulled from the statement and shares its interview with Ron Wasserstein, the ASA’s executive director.

Perhaps chief among the six principles is the third: “Scientific conclusions and business or policy decisions should not be based only on whether a p-value passes a specific threshold.”

Beyond the misunderstandings, Wasserstein was troubled by devious applications of the statistic.

“What concerns us even more are the misuses, particularly the misuse of statistical significance as an arbiter of scientific validity. Such misuse contributes to poor decision making and lack of reproducibility and ultimately erodes not only the advance of science but also public confidence in science.”

References

1. Baker M. Statisticians issue warning over misuse of P values: Policy statement aims to halt missteps in the quest for certainty. Nature 531(7593): 151, 2016. Available here.
2. Wasserstein RL and Lazar NA. The ASA’s statement on p-values: context, process, and purpose. The American Statistician, 2016. Available here.