Migrating Data Ingestion Systems at Meta Scale

Meta's engineering team shares how they migrated their entire data ingestion system — which scrapes petabytes of social graph data from MySQL into the data warehouse daily — from a legacy customer-owned pipeline architecture to a new self-managed service. The migration covered tens of thousands of jobs and involved a three-phase lifecycle: shadow testing in pre-production, a reverse shadow phase for early rollout signals and fast rollback, and cleanup. Key strategies included automated tooling to monitor and promote jobs through lifecycle stages, custom data quality analysis comparing row counts and checksums, CDC-aware rollback mechanisms to stop bad data propagation, and batched migration with capacity planning to avoid unnecessary full dumps.