Survivorship Bias

The Statistical Research Group, where Abraham Wald spent much of World War II, was a classified program that yoked the assembled might of American statisticians to the war effort:  something like the Manhattan Project, except the weapons being developed were equations, not explosives. The military came to the SRG with some data they thought might be useful. When American planes came back from engagements over Europe, they were covered in bullet holes. But the damage wasn’t uniformly distributed across the aircraft. There were more bullet holes in the fuselage, not so many in the engines. The officers saw an opportunity for efficiency; you can get the same protection with less armor if you concentrate the armor on the places with the greatest need, where the planes are getting hit the most. But exactly how much more armor belonged on those parts of the plane? That was the answer they came to Wald for. It wasn’t the answer they got.
The armor, said Wald, doesn’t go where the bullet holes are. It goes where the bullet holes aren’t: on the engines. Wald’s insight was simply to ask: where are the missing holes? The ones that would have been all over the engine casing, if the damage had been spread equally all over the plane? Wald was pretty sure he knew. The missing bullet holes were on the missing planes. The reason planes were coming back with fewer hits to the engine is that planes that got hit in the engine weren’t coming back. Whereas the large number of planes returning to base with a thoroughly Swiss-cheesed fuselage is pretty strong evidence that hits to the fuselage can (and therefore should) be tolerated. To a mathematician, the structure underlying the bullet hole problem is a phenomenon called survivorship bias.

Inspired by: Jordan Ellenberg (2015). How Not to Be Wrong: The Power of Mathematical Thinking, Penguin Books.

Leave a Reply