Preventing Revenue Loss With Real-Time A/B Test Monitoring

Expedia Group built a real-time A/B test monitoring system called EGTnL Circuit Breaker using Apache Flink to prevent revenue loss from poorly performing experiments. The system addresses the critical gap in monitoring during the first 24 hours of an experiment by aggregating user exposures and metrics in real-time, handling complex challenges like bot traffic filtering and distinct user tracking. The solution processes data through three stages: filtering, user state collection, and aggregation, enabling automatic detection and suspension of underperforming tests within minutes. In its first 6 months, the system monitored 95% of A/B tests, detected 36% of experiment issues automatically, and prevented significant revenue loss including stopping one experiment with a -39% conversion impact.