Not all multiple tests require the same statistical adjustments for significance

Not all multiple tests require the same statistical adjustments for significance
Image: Intellect (generated)

In short: Researchers often adjust their significance thresholds when performing multiple comparisons to avoid false positives, a practice central to addressing the replication crisis. However, this paper argues that these adjustments are not universally applicable. By categorizing testing into three distinct types—disjunction, conjunction, and individual—the authors found that standard alpha adjustments are only appropriate for specific logical structures of scientific inquiry.

The distinction matters because applying the wrong adjustment can lead to incorrect conclusions about a study's results. For example, while disjunction testing (where at least one result must be significant) requires adjustment, conjunction testing (requiring all results to be significant) and individual testing do not.

The paper analyzes these logical conditions through examples like joint multiway ANOVA hypotheses. A primary limitation is that the study provides a theoretical framework for when to adjust alpha rather than providing new empirical data from specific experiments.

Why it matters: Understanding the logical structure of your research question determines whether you need to adjust your statistical significance thresholds.

Opens the Intellect+ feed with this card first. Swipe up for the next one.

Sources

To cite this card, link to https://intellectplus.co.uk/app/science-research-not-all-multiple-tests-require-the-same-statistical-adjustme.

How we source, check and correct cards: Editorial standards.

General information, not professional advice. Spotted an error? Flag the card or see corrections and complaints.

Intellect Plus is a visual educational discovery platform for exploring fascinating facts and ideas. Brain Food, Not Brain Rot. What is Intellect Plus?

Related

Finding something worth knowing…