Why a 'significant' result might actually be barely worth mentioning
In short: In frequentist statistics, a p-value below 0.05 is often treated as a victory. But Bayesian analysis reveals a deeper truth: a result can be statistically significant while still providing almost no evidence for a theory. The Bayes factor exposes this nuance by weighing competing models against one another.
The Bayes factor (K) is the ratio of two marginal likelihoods, representing the support one statistical model has over another. Unlike frequentist null hypothesis significance testing, which only determines if the null can be rejected, the Bayes factor can actively quantify evidence in favor of the null. This makes it a more complete tool for model selection, as it evaluates the plausibility of two competing hypotheses based on observed data.
A key advantage of the Bayes factor is its inherent 'Occam's razor' effect. It naturally penalizes models with excessive complexity or too many free parameters, guarding against overfitting. For example, in a study comparing a model where a success probability is fixed at 0.5 against one where it is unknown, a frequentist test might find a result 'significant' due to a small sample size. However, the Bayes factor might show the result is 'barely worth mentioning' because the second model's extra flexibility doesn't sufficiently explain the data over the simpler null.
Calculating these factors is often computationally difficult because closed-form expressions for marginal likelihoods are rarely available. Researchers frequently rely on numerical approximations like MCMC samples or the Bayesian information criterion (BIC). Because these approximations can be biased—especially in small datasets where priors matter—modern procedures like marginal simulation-based calibration (SBC) are used to verify that the calculated Bayes factor is accurate. If these calibration checks fail, the resulting evidence cannot be trusted.
To interpret the strength of evidence, researchers often use Jeffreys' scale or the Kass and Raftery scale. These provide a framework for converting the ratio into 'decibans' or 'bits,' allowing scientists to communicate exactly how much more plausible one model is than another. This shift from 'reject or accept' to 'how much evidence' provides a more nuanced view of scientific discovery.
Opens the Intellect+ feed with this card first. Swipe up for the next one.
Sources
- Summary by Intellect · source: Bayes factor
- Also: How accurate are Bayes factor-based null hypothesis tests? A simulation study.
- Also: Test (Madrid, Spain) — Analytic posterior distribution and bayes factor for pearson partial correlations. · CC BY
To cite this card, link to https://intellectplus.co.uk/app/health-bayes-factor.
How we source, check and correct cards: Editorial standards.
General information, not professional advice. Spotted an error? Flag the card or see corrections and complaints.
Intellect Plus is a visual educational discovery platform for exploring fascinating facts and ideas. Brain Food, Not Brain Rot. What is Intellect Plus?
Related
More in Epidemiology · More in Health · Latest · A–Z · All topics
