Asana's platform engineering team scaled their invalidation pipeline to handle near-realtime reactivity for millions of users. The pipeline uses change-data-capture from MySQL binlog to decouple mutations from invalidations, automatically determining affected data and notifying clients. The first major scaling challenge was expensive transactions causing head-of-line blocking. The solution involved introducing out-of-order invalidation delivery with a DataFreshness construct (similar to vector clocks) that maintains eventual consistency while allowing slow transactions to be processed asynchronously. This approach eliminated manual intervention and has worked successfully for 6 years without further optimization.
Table of contents
How Asana Keeps Data in SyncPubsubInvalidation pipelineWhy this design?Scaling challengesHow did we actually solve these problems?Problem 1: Expensive transactions and head-of-line blockingWhat breaks next?Sort: