The most important data is often the data that never came back.
In the Second World War, statisticians studied bullet damage on planes returning from missions. The catch: the planes that were shot down never came back to be counted. Survivorship bias, looking only at what made it through a filter, quietly distorts everything from investment returns to success stories.
Survivorship bias is the error of drawing conclusions from the people, companies or things that passed some selection process while ignoring those that did not. Because the failures are invisible, the survivors can look more typical, or more special, than they really are.
The classic case comes from the Statistical Research Group at Columbia University during the Second World War. The statistician Abraham Wald, one of its members, worked out methods for estimating how vulnerable different parts of an aircraft were, using damage on planes that had come back. The difficulty was that the sample was filtered: holes on returning planes show where an aircraft can be hit and still fly home. Wald suggested his results could guide where to place protective armour, and the work is regarded as a foundation of operations research.
The same trap appears everywhere. Investment funds that perform badly are often closed or merged, so a list of today's funds looks healthier than the real record. A 1987 study found that cats falling from more than six storeys seemed to suffer fewer injuries than those falling from lower down; one proposed explanation is that cats killed in high falls were rarely taken to a vet, so they never entered the data. A widely reported study claiming Oscar winners lived almost four years longer than other actors shrank to about a year, and lost statistical significance, when reanalysed to fix a related timing error.
In science, the bias feeds publication bias: if many labs test an idea, some will get striking results by chance, and those are the ones most likely to be published. In everyday life it shapes the stories we hear about dropouts turned billionaires, while the many equally determined people who failed go untold. The useful habit is to always ask: what am I not seeing?
Source: Wikipedia — Survivorship bias · Text summarised from Wikipedia (CC BY-SA 4.0)
