Skip to Content

We’re Using a Common Statistical Test All Wrong. Statisticians Want to Fix That.

ByCrossFitFebruary 16, 2019

P-values, or probability values, are  “commonly used to test (and dismiss) a ‘null hypothesis’, which generally states that there is no difference between two groups, or that there is no correlation between a pair of characteristics. The smaller the P value, the less likely an observed set of values would occur by chance — assuming that the null hypothesis is true. A P value of 0.05 or less is generally taken to mean that a finding is statistically significant and warrants publication.” (1)

Moved by growing concerns about the reproducibility of research, the American Statistical Association (ASA) issued a statement in March 2016 to address the widespread misuse of p-values. ¹

In this article, Retraction Watch supplies a set of six principles pulled from the statement and shares its interview with Ron Wasserstein, the ASA’s executive director.

The widespread use of ‘statistical significance’ (generally interpreted as ‘p & 0.05’) as a license for making a claim of a scientific finding (or implied truth) leads to considerable distortion of the scientific process.

ASA Statement on Statistical Significance and P-values, on behalf of the American Statistical Association Board of Directors

Perhaps chief among the six principles is the third: “Scientific conclusions and business or policy decisions should not be based only on whether a p-value passes a specific threshold.”

Beyond the misunderstandings, Wasserstein was troubled by devious applications of the statistic.

“What concerns us even more are the misuses, particularly the misuse of statistical significance as an arbiter of scientific validity. Such misuse contributes to poor decision making and lack of reproducibility and ultimately erodes not only the advance of science but also public confidence in science.”


References

  1. Baker M. Statisticians issue warning over misuse of P values: Policy statement aims to halt missteps in the quest for certainty. Nature 531(7593): 151, 2016. Available here.
  2. Wasserstein RL and Lazar NA. The ASA’s statement on p-values: context, process, and purpose. The American Statistician, 2016. Available here.

Comments on We’re Using a Common Statistical Test All Wrong. Statisticians Want to Fix That.

Comments

Comment thread URL copied!