How Pinterest Built An Async Compute Platform for Billions of Task Executions

Pinterest rebuilt their asynchronous job processing platform from Pinlater to Pacer to handle billions of daily tasks. The original system suffered from database lock contention, lack of queue isolation, and inefficient sharding. Pacer introduced dedicated dequeue broker services managed by Helix, eliminated lock contention through single-broker partition ownership, implemented in-memory caching for sub-millisecond latency, adopted adaptive sharding based on queue size, and isolated worker pods on Kubernetes with custom resource allocations per queue.