Many years ago, I used to think that A/B tests were foolproof and all you need to do is compare the metrics for the two groups. The group with the highest conversion rate wins, right? Then, for a long period, I ran a lot of tests.

Erik Bernhardsson

A pushback against the perfectionist gatekeeping around A/B testing. The core argument is that even imperfect A/B testing is beneficial for product teams because the stakes are fundamentally different from drug trials: false positives are cheap (a wrong button color), prior success rates are higher (tests are hypothesis-driven), and true positives have real business value. The practical takeaway is to pick the metric with the best signal-to-noise ratio before running the test, avoid ratio metrics, and not cherry-pick metrics after the fact. More A/B testing, even imperfect, beats paralysis from over-theorizing.

In defense of false positives (why you can't fail with A/B tests)