This is going to be an embarrassing story of how I messed up over 2 million records in production, almost causing our Kubernetes pods to crash. But I think it's a good learning experience that other engineers can learn from.

Timeline:
00:00:00 - Intro
00:00:11 - Background
00:00:55 - What my team was working on
00:01:33 - What is Mongo CSFLE
00:02:19 - Migration strategy 
00:03:42 - What went wrong
00:06:15 - Solution
00:07:10 - Ending

Credits:

Music credit: - ♪ [non copyright music] Lofi Type Beat - Saturday Morning | aesthetic lofi music / Lofiru Link :    [![](https://www.gstatic.com/youtube/img/watch/yt_favicon.png) • [non copyright music] Lofi Type Beat ...](https://www.youtube.com/watch?v=LEMZctXCN7Q&t=0s)

Kiki's Bytes 

A migration script to encrypt sensitive user data in MongoDB corrupted over 2 million records by accidentally wiping out unencrypted fields. The bug occurred because MongoDB's $set operation replaces entire nested objects rather than merging properties. Running the migration twice resulted in both encrypted and unencrypted fields being set to undefined, causing queries to return millions of records and spike memory usage in Kubernetes pods. Recovery required restoring a database backup from before the migration and cross-referencing with Redshift data to identify users who registered during the incident window.

How I Corrupted 2 Million+ DB Records in Prod

wow, thanks for sharing, sounds like a nightmare.
i guess we cannot avoid these mistakes 100% unless not doing the work.
Those who do things will also go wrong.
The trick, like you mentioned, is to own our mistakes, take responsibility and to use it as opportunity to learn and grow.

When I read the title, I was terrified, and I was like: “This guy is cooked”, but tis kinda fun isn’t it? Like, in future, whenever you will remember your best memories, you’d remember: “Once, I corrupted 2 million db records, lol”

I work a lot with SQL server and do a lot of backend work. I made quite a few mistakes, and broke a couple of DB’s myself, before I got to workable process to eliminate most of the errors I made.
<ol>
<li>Always test on a QA enviroment first.</li>
<li>Read your sql scripts 7 times. It takes longer to fix an error than to read that script 7 times.</li>
<li>Always have a rollback script for the work to be done. If there is an error you can easily revert and fix.</li>
<li>If you are tired of testing, you know you are ready to push to production.</li>
<li>Even if you do all these things above you will still make mistakes as there is a built in complexity in modern production db’s and servers, like you mentioned above.</li>
<li>Document your work and change process, if there is an investigation afterward chances are you will get off more easily as people can see you put in the precautionary work.</li>
</ol>

Wow. Just… Wow.
I think the mistake was made much earlier on by letting someone without the basic knowledge of MongoDB fundamentals anywhere near a production database.