Statistical significance is not enough: why effect size matters more than p-values
In short: A p-value tells you if a result is likely due to chance, but it says nothing about the actual importance of the finding. To understand the real-world magnitude of a phenomenon, you need an effect size. It is the essential metric for separating trivial statistical noise from meaningful discovery.
In the world of empirical research, statistical significance often masquerades as importance. However, with a sufficiently large sample size, even a tiny, practically useless relationship can produce a statistically significant p-value. For instance, a Pearson correlation of 0.01 becomes statistically significant with a sample of 1,000, yet such a weak correlation is often meaningless in practice. Effect size fills this gap by quantifying the magnitude of a phenomenon, providing a necessary complement to hypothesis testing.
Effect sizes come in two primary flavors: unstandardized and standardized. Unstandardized measures, like mean differences or regression coefficients, are preferred when the units of measurement—such as cigarettes smoked per day—have inherent meaning. Standardized measures, such as Cohen’s d or Pearson’s r, are used when variables lack intrinsic meaning or when researchers need to compare results across different studies using different scales. By expressing the effect relative to population variability, standardized measures allow for the synthesis of knowledge in meta-analyses.
While Jacob Cohen provided ubiquitous guidelines for interpreting effect sizes as small, medium, or large, these conventions are increasingly viewed as problematic. Critics argue that such labels are arbitrary and ignore the specific context of the research field. A 'small' effect in one domain might be monumental in another. Modern best practices suggest grounding results in meaningful, field-specific benchmarks rather than relying on universal rules of thumb. Ultimately, effect size is a tool for interpretation, not a substitute for critical thinking about the substantive contribution of a research result.
The calculation of these metrics is sensitive to sampling error and bias. Small studies, in particular, are prone to the 'small-study effect,' where reported effect sizes are often inflated, potentially signaling publication bias. Because effect sizes are used to weight studies in meta-analyses, understanding the uncertainty—often represented by standard deviation—is critical. Without accounting for this uncertainty, researchers risk overestimating the true population effect, leading to a distorted view of scientific evidence.
Opens the Intellect+ feed with this card first. Swipe up for the next one.
Sources
- Summary by Intellect · source: Effect size
To cite this card, link to https://intellectplus.co.uk/app/science-effect-size.
How we source, check and correct cards: Editorial standards.
General information, not professional advice. Spotted an error? Flag the card or see corrections and complaints.
Intellect Plus is a visual educational discovery platform for exploring fascinating facts and ideas. Brain Food, Not Brain Rot. What is Intellect Plus?
Related
More in Statistics and probability · More in Science · Latest · A–Z · All topics