Experiment Design: Buy Answers, Not Rituals

Two teams both say they "run experiments."

Team A's log: "Tried a referral program — engagement felt better. Tested new onboarding — people seemed to like it. Ran a survey — mostly positive!" Twelve entries, twelve vaguely warm feelings, zero decisions changed.

Team B's log has four entries. Each one reads like a bet: "We believe churned users leave because setup takes 40 minutes. If we cut it to 10 for half of new signups, week-2 retention in that group rises by 15% or more. It didn't — 3% lift. Killed the setup rewrite, went digging in exit interviews instead."

Team B looks slower. Team B has learned more in four experiments than Team A in twelve.