How Wrong Can You Get?

From an article in Harvard Business Review, April 2005, "Selection Bias and the Perils of Benchmarking".

During World War II, the statistician Abraham Wald was assessing the vulnerability of airplanes to enemy fire. All the available data showed that some parts of planes were hit disproportionately more often than other parts.

Military personnel concluded, naturally enough, that these parts should be reinforced. Wald, however, came to the opposite conclusion: The parts hit least often should be protected.

His recommendation reflected his insight into the selection bias inherent in the data, which represented only those planes that returned.

Wald reasoned that a plane would be less likely to return if it were hit in a critical area and, conversely that a plane that did return had probably not been hit in a critical location. Thus, he argued, reinforcing those parts of the returned planes that sustained many hits would be unlikely to pay off.

Selection bias it would seem is something we all need to consider when undertaking any form of statistical analysis during our daily work.

Note: The Wald story is one of the most widely cited anecdotes in the statistical community. To find out more about it see W. Allen Wallis, "The Statistical Research Group, 1942-1945", Journal of the American Statistical Association, June 1980, and M. Mangel & F.J. Samaniego, "Abraham Wald's work on Aircraft Survivability", Journal of the American Statistical Association, June 1984.