Kiki's Bytes 

A migration script to encrypt sensitive user data in MongoDB corrupted over 2 million records by accidentally wiping out unencrypted fields. The bug occurred because MongoDB's $set operation replaces entire nested objects rather than merging properties. Running the migration twice resulted in both encrypted and unencrypted fields being set to undefined, causing queries to return millions of records and spike memory usage in Kubernetes pods. Recovery required restoring a database backup from before the migration and cross-referencing with Redshift data to identify users who registered during the incident window.

How I Corrupted 2 Million+ DB Records in Prod

Your go-to source for the latest in database technology, trends, and tips. Covering Postgres, MySQL, Kafka, Data Warehouses, and more. 