Your order service publishes 3 events per order: created, paid, cancelled.
Standard SQS queue, 5 consumers, ~2K orders/minute at peak.
Last night, prod alerted: 47 orders stuck in “cancelled” state with no “created” record. Customers got refunded for orders that were never marked as placed. Finance is not happy.
You dig in. The order was placed, paid, and cancelled within 400ms. Three messages hit SQS in order. Three consumers picked them up in parallel. “cancelled” got processed first. State machine rejected “created” as invalid. Data is now wrong.
Here’s the setup:
OrderService → SQS (standard) → 5 workers → Postgres
Events: order.created → order.paid → order.cancelled
Problem: workers process in parallel, SQS standard doesn’t guarantee order, downstream state machine breaks when events arrive out of sequence.
You have until Monday. What do you do?
A) Switch to SQS FIFO + MessageGroupId per order_id — let AWS handle ordering per order.
B) Keep standard SQS, add a sequence number + reorder buffer in the consumer — hold out-of-order events until the earlier ones arrive.
C) Replace the event stream with a Saga — each step waits for the previous step’s completion signal before firing the next.
D) Add an event version/timestamp and make the state machine idempotent + order-agnostic — reject stale transitions, accept any arrival order.
All four are patterns I’ve seen in real production systems. Only one actually fits this problem. One of them is the trap that gets senior engineers on the whiteboard.
Pick one — A, B, C, or D — and tell me why. Full breakdown in the comments.
If your team has argued about ordering guarantees at 2am, forward this to them. That debate is where the real learning happens.
Drop your answer 👇
#30DaysOfSystemDesign #Day7 #SystemDesign #DistributedSystems

Joud Awad

A production incident scenario where 47 orders got stuck in a 'cancelled' state with no 'created' record, caused by out-of-order message processing in a standard SQS queue with 5 parallel consumers. Three events (created, paid, cancelled) fired within 400ms were picked up by different workers simultaneously, causing the downstream state machine to process 'cancelled' before 'created'. Four solution patterns are presented: SQS FIFO with MessageGroupId, a reorder buffer in the consumer, a Saga pattern, or making the state machine idempotent and order-agnostic. Readers are asked to pick the best solution and explain their reasoning.

7/30 Days System Design Questions!

✅ Answer: A — SQS FIFO + MessageGroupId per order_id
Why A wins:
You don’t need global ordering — you need per-order ordering. Set MessageGroupId = order_id. FIFO guarantees messages with the same group ID are delivered in exact send order, with only one consumer processing that group at a time. Your 2K orders/min still fan out across 5 workers — throughput is unchanged. Order #12345’s created → paid → cancelled always lands in sequence. Cost: a few ms latency. Ship it Monday.
<img src="https://media.daily.dev/image/upload/s--xtCNxGTa--/f_auto/v1778601793/ugc/content_3b01b1ed-2c1a-4041-a248-572d89607329?_a=BAMAMiWQ0" alt="ChatGPT Image May 12, 2026, 07_01_58 PM (1).png">

Why B is the trap (reorder buffer):
Fools senior engineers on the whiteboard. You’d end up building: in-memory buffer per order_id + TTL logic + dead-letter path for gaps + crash recovery + stuck buffer monitoring. You just rebuilt FIFO in app code. With more bugs. And you own every edge case. B is valid only when you can’t use FIFO (e.g. Kafka with a taken partition key). That’s not this situation.

Why C is wrong (Saga):
Saga is for long-running distributed transactions with compensating actions (book flight → charge card → reserve hotel). Your problem is a state stream for one entity — not a multi-service transaction. Forcing it into a Saga turns async events into sync RPC, increases coupling, and still doesn’t fix ordering.

Why D is wrong (event versioning):
Partially works — but you’d drop legitimate events. “cancelled” arrives first, moves state forward, then “created” gets rejected. DB says order is cancelled with no creation record. To make D work correctly you need full event sourcing (6-month rewrite, not a Monday fix).

Also, it would mean a lot to me if you could support my content and stay in touch 🙏
<ul>
<li>YouTube: <a href="https://www.youtube.com/@system-design-lab" target="_blank" rel="noopener nofollow">https://www.youtube.com/@system-design-lab</a></li>
<li>LinkedIn: <a href="https://www.linkedin.com/in/joud-awad/" target="_blank" rel="noopener nofollow">https://www.linkedin.com/in/joud-awad/</a></li>
<li>Medium Blog: <a href="https://joudwawad.medium.com/" target="_blank" rel="noopener nofollow">https://joudwawad.medium.com/</a></li>
<li>Substack: <a href="https://joudawad.substack.com/" target="_blank" rel="noopener nofollow">https://joudawad.substack.com/</a></li>
</ul>