Trusting the Untestable: Validation and Diagnostics for the Doubly Robust Models

Lyft's data science team developed validation methods for Augmented Inverse Propensity Weighting (AIPW), a doubly robust causal inference model used when A/B testing isn't feasible. The platform requires rigorous confounder management with hundreds of features, applies propensity score corrections for downsampled data, and provides diagnostic scorecards checking propensity overlap and covariate balance. Validation against experimental ground truth from ride challenge programs revealed AIPW understates effects by 16% due to propensity trimming creating non-representative samples. The team added marginal sensitivity models and covariate comparison diagnostics to detect when hidden confounders or trimming compromise estimate reliability.