In technical products, there’s a tendency to lean towards A/B tests. To run simultaneous changes across different slices of your user base and to measure the outcome.

A/B tests can be extremely useful in some cases — if you’re at Google or Meta scale or if you’re doing something like performance marketing. But in the vast majority of cases, it’s more pain than it’s worth — and might even be detrimental.

 1. You don’t have enough data. Most products don’t have enough users to generate statistic

MattRickard's platform  covers topics related to software development, technology trends, and coding tutorials. Through articles, tutorials, and personal anecdotes, MattRickard offers insights into programming languages, development frameworks, and best practices in software engineering. Readers can learn about software design principles, development methodologies, and emerging technologies to enhance their coding skills and build high-quality software applications.

Matt Rickard

A/B testing may not be effective for most products as they often lack sufficient data, lead to incremental changes, require double the work, and may not have clear goals. Startups, in particular, need to prioritize and may not have the resources for extensive testing.

When A/B Testing Doesn't Work