In September 2025, engineers working on the Matrix.org homeserver proved once more that technological progression will never save engineers from manually deleting their production database.

Sources:
https://matrix.org/blog/2025/10/post-mortem/
https://github.com/wal-g/wal-g/blob/master/internal/databases/postgres/backup.go

Chapters:
0:00 - intro
0:45 - matrix
2:14 - the databases
3:30 - the Incident
7:53 - the deletion
9:12 - the recovery
11:32 - how to get good

Music:
Finding the Balance by Kevin MacLeod: https://www.youtube.com/watch?v=6FthX-mZFNo

Twitter: https://twitter.com/kevinfaang/
Instagram: https://instagram.com/kevinfaang_yt/

Kevin Fang

Element Creations experienced a 24-hour outage of the matrix.org home server after an engineer accidentally deleted the production database while attempting to restore a failed server. The incident began with a hardware failure requiring database migration, but confusion over which server was primary led to running a destructive command on the wrong machine. Recovery took over a day due to slow backup restoration (51TB), a bug in their backup tool that wasn't patched in production, and slow write-ahead log replay. The postmortem emphasizes faster backup restoration strategies, including local snapshots using copy-on-write filesystems like ZFS, and highlights how operational errors during high-pressure situations are nearly inevitable.

Dev Picks the Wrong Database, Takes Down Company

<p>Are you even a real dev if you haven’t nuked production at some point. It’s the mistakes that make us better</p>


<p>Is this really that easy to accidentally delete databases in prod?
I mean I’ve seen bunch of such cases till date!</p>
