3:17 AM. One payment service started responding in 8 seconds instead of 80ms.
By 3:19, six services were down. Not the payment service. Six completely different ones.
Here’s what happened.
Payment had 12 upstream callers. Every one of them had the same “resilient” retry logic: 3 attempts, 2-second backoff. On paper, sensible. In production, it multiplied the load by 4x in under a minute.
At 3:17 payment was slow, not dead. 12 callers timed out. Each caller kicked off 3 retries. The first retries arrived while the original requests were still in flight. Payment’s thread pool went from 80% utilized to 100%, with a queue of duplicates stacked on top.
At 3:18 the retries started timing out too. Now every caller was in its own retry loop, and every caller had its own upstream callers doing the same thing. Checkout retried against orders. Orders retried against inventory. Inventory retried against pricing. The retry storm walked up the call graph in 90 seconds.
At 3:19 six services were unresponsive. Not because they were broken — because they were drowning in retried requests from services that couldn’t tell the difference between “slow” and “dead.”
The fix wasn’t more retries. It was fewer.
A circuit breaker in front of payment would have opened after 5 failures, rejected the next 10 seconds of traffic instantly, and given payment room to breathe. The callers would have failed fast, returned a cached response or a degraded path, and stopped piling on. One service would have gone down. Not six.
The rule nobody writes down: retries without a circuit breaker is just a DDoS attack you wrote against yourself.
Retries assume the downstream is healthy and hit a transient blip. Circuit breakers assume the downstream is sick and needs quiet. You need both, in that order: circuit breaker first, retry inside it. Never retry through an open circuit.
The fix:
• Wrapped every outbound call in a circuit breaker (opossum for Node, resilience4j for Java)
• Retries allowed only inside a closed circuit, with backoff jitter
• Capped retry budgets per request chain to prevent load multiplication
Zero cascades since.
If this saved you a 3am postmortem, repost it — someone on your on-call rotation needs this before their next outage.
One question: when your payment service goes slow, do your callers retry first or fail fast first? There’s only one right answer, and most teams have it wrong.
#SystemDesign #DistributedSystems #Microservices #BackendEngineering #SoftwareEngineering

You can share you thought with this squad 

Backend Developer 

A real-world incident postmortem where a single slow payment service triggered a cascading failure across six unrelated services in under two minutes. The root cause was retry logic without circuit breakers — each of 12 upstream callers retried 3 times with backoff, multiplying load 4x and creating a self-inflicted DDoS. The fix: wrap every outbound call in a circuit breaker (opossum for Node.js, resilience4j for Java), only allow retries inside a closed circuit with jitter, and cap retry budgets per request chain. The key principle: circuit breaker first, retry inside it — never retry through an open circuit.

Circuit Breaker Explained