Short practical advice on A/B testing: Stop sizing tests only for statistical significance - In finite campaigns, your goal is profit, not perfect inference. Treat testing as a trade-off - Every extra test exposure buys learning but ...

R-bloggers

A review of the 'Test & Roll: Profit-Maximizing A/B Tests' paper by Feit and Berman (2019), arguing that classical hypothesis-test sizing is wrong for finite marketing campaigns. The key insight: when you have a limited audience, the right objective is total expected profit across test and rollout stages, not p-values. Profit-maximizing test sizes grow sub-linearly with noise and scale with the square root of population size, making them much smaller than classical power-based sizes. The post also covers when unequal splits are optimal, how this compares to multi-armed bandits, and provides a practical implementation checklist for email, paid media, and landing page tests.

Test & Roll: Why Smaller A/B Tests Can Make More Money