When I told 4,091 writers they weren’t getting paid

On September 5, 2024, a new Partner Program payments system was turned on, leading to an incident where 4,091 writers were wrongly informed they wouldn't be paid due to a subtle database race condition. The problem stemmed from the interaction between write and read operations on the database, exacerbated by the use of read replicas. The issue was mitigated by adjusting the sequence of database operations. Lessons were learned about managing race conditions and the necessity of simulating production environments during testing.