Your Kafka consumer is processing 800 events/sec.
The producer just hit 5,000 events/sec and it’s not slowing down.
Lag chart: 12 minutes behind and climbing. Consumer memory: 89% and rising. The on-call alert just fired. You have ~4 minutes before the JVM starts GC-thrashing and the pod gets OOM-killed.
Here’s the setup:
Producer → Kafka topic (5K events/sec, growing)
Consumer → spring-kafka @KafkaListener, batch=500, processes ~800 events/sec
Downstream → Postgres write + external HTTP call (the real bottleneck)
SLA → events must be processed, not silently dropped
The consumer can’t keep up. The producer doesn’t know it. What do you do?
A) Drop events on the floor — fail fast, return early, let the lag burn down. The system stays alive.
B) Block the producer — make the consumer signal “slow down,” apply backpressure upstream until it catches up.
C) Buffer harder — bigger in-memory queue, larger batch size, scale the consumer to absorb the spike.
D) Rate-limit + load-shed — cap consumption rate, route the overflow to a DLQ or secondary topic for later replay.
Three of these are real production patterns. Only one of them actually fits this stack and this SLA.
Pick one — A, B, C, or D — and tell me why. Full breakdown in the comments (including why two of the wrong answers will fool engineers who’ve shipped Kafka before).
If your team argues about backpressure in standups, share this with them. The right answer is platform-specific, and most posts get it wrong.
Drop your answer 👇
#30DaysOfSystemDesign #Day17 #SystemDesign #DistributedSystems

Smiley Face

Joud Awad

A real-world Kafka consumer lag scenario where a consumer processing 800 events/sec is overwhelmed by a producer pushing 5,000 events/sec. With memory at 89% and an OOM kill imminent in ~4 minutes, readers must choose between four strategies: dropping events, blocking the producer with backpressure, buffering harder, or rate-limiting with load shedding to a DLQ. The post frames this as a system design challenge where the correct answer depends on the specific stack (Spring Kafka + Postgres + external HTTP) and the SLA requirement that events must not be silently dropped.

17/30 Days System Design Questions!

what if the events are state dependent then you can’t load it to DLQ to process it later

What is the difference between D vs scaling the consumer? Is there any reason that we don’t just scale the consumer rather than having another topic

Very good.
I am learning from here regularly.
Thank you.

This was a really good scenario to learn from

lso, it would mean a lot to me if you could support my content and stay in touch 🙏
<ul>
<li>YouTube: <a href="https://www.youtube.com/@system-design-lab" target="_blank" rel="noopener nofollow">https://www.youtube.com/@system-design-lab</a></li>
<li>LinkedIn: <a href="https://www.linkedin.com/in/joud-awad/" target="_blank" rel="noopener nofollow">https://www.linkedin.com/in/joud-awad/</a></li>
<li>Medium Blog: <a href="https://joudwawad.medium.com/" target="_blank" rel="noopener nofollow">https://joudwawad.medium.com/</a></li>
<li>Substack: <a href="https://joudawad.substack.com/" target="_blank" rel="noopener nofollow">https://joudawad.substack.com/</a></li>
</ul>