Why the scientific gold standard of statistical significance is under fire
In short: For nearly a century, researchers have used p-values to decide if a discovery is real. But this threshold, often set at 0.05, has created an illusion of certainty, fueling reproducibility crises and distorting scientific claims. Now, many experts are calling to abandon the term entirely in favor of more nuanced analysis.
Statistical significance is a tool used in hypothesis testing to determine if an observed result is likely to be real or merely a product of sampling error. A researcher calculates a p-value—the probability of obtaining a result as extreme as the one observed, assuming the null hypothesis (that no effect exists) is true. If this p-value falls below a pre-specified significance level, typically 0.05, the result is deemed statistically significant. This framework, popularized by Ronald Fisher in 1925, was intended to be flexible, but it has hardened into a rigid gatekeeper for scientific publication.
The reliance on this binary threshold has drawn intense criticism. In 2016, the American Statistical Association warned that using p-values as a license to claim a scientific finding distorts the research process. Critics argue that statistical significance is frequently confused with practical or clinical importance. A study might show a statistically significant effect that is nonetheless weak, irrelevant, or impossible to replicate. Because failing to replicate a result increases the likelihood that the original finding was a false positive, the current system may be actively misleading the scientific community.
Reformers suggest several paths forward. Some propose lowering the threshold to 0.005 to reduce false positives, though others warn this could increase false negatives or encourage data dredging. More radical voices, including hundreds of scientists and statisticians, advocate for abandoning the term statistically significant altogether. They argue for treating p-values as continuous indices or employing alternative measures like Bayes factors. Ultimately, the goal is to shift focus from arbitrary cutoffs toward a more rigorous, transparent evaluation of evidence that acknowledges the complexity of scientific discovery.
Opens the Intellect+ feed with this card first. Swipe up for the next one.
Sources
- Summary by Intellect · source: Statistical significance
To cite this card, link to https://intellectplus.co.uk/app/technology-statistical-significance.
How we source, check and correct cards: Editorial standards.
General information, not professional advice. Spotted an error? Flag the card or see corrections and complaints.
Intellect Plus is a visual educational discovery platform for exploring fascinating facts and ideas. Brain Food, Not Brain Rot. What is Intellect Plus?
Related
More in Motorbikes · More in Technology · Latest · A–Z · All topics