Only the planes that came back
Survivorship bias is easy to spot in a history lesson and a lot harder to spot in your own data.
The way the story usually gets told: during World War II, analysts looked at bombers coming back full of bullet holes and wanted to add armor where they were hit. The statistician Abraham Wald pointed out that the planes hit in other places were the ones that never made it back, so the clean areas were the ones worth protecting.
A customer survey can work the same way. If you only survey current customers, you’re mostly hearing from people the product already works for. The ones who left, possibly for the exact reason you’d most want to know about, aren’t in the results, and a strong satisfaction score can quietly hide that.
That doesn’t make the survey useless. I think it means asking who isn’t in the data before deciding what the data says.
Where have you seen a decision made from only the planes that came back?
