Your order service publishes 3 events per order: created, paid, cancelled.
Standard SQS queue, 5 consumers, ~2K orders/minute at peak.
Last night, prod alerted: 47 orders stuck in “cancelled” state with no “created” record. Customers got refunded for orders that were never marked as placed. Finance is not happy.
You dig in. The order was placed, paid, and cancelled within 400ms. Three messages hit SQS in order. Three consumers picked them up in parallel. “cancelled” got processed first. State machine rejected “created” as invalid. Data is now wrong.
Here’s the setup:
OrderService → SQS (standard) → 5 workers → Postgres
Events: order.created → order.paid → order.cancelled
Problem: workers process in parallel, SQS standard doesn’t guarantee order, downstream state machine breaks when events arrive out of sequence.
You have until Monday. What do you do?
A) Switch to SQS FIFO + MessageGroupId per order_id — let AWS handle ordering per order.
B) Keep standard SQS, add a sequence number + reorder buffer in the consumer — hold out-of-order events until the earlier ones arrive.
C) Replace the event stream with a Saga — each step waits for the previous step’s completion signal before firing the next.
D) Add an event version/timestamp and make the state machine idempotent + order-agnostic — reject stale transitions, accept any arrival order.
All four are patterns I’ve seen in real production systems. Only one actually fits this problem. One of them is the trap that gets senior engineers on the whiteboard.
Pick one — A, B, C, or D — and tell me why. Full breakdown in the comments.
If your team has argued about ordering guarantees at 2am, forward this to them. That debate is where the real learning happens.
Drop your answer 👇
#30DaysOfSystemDesign #Day7 #SystemDesign #DistributedSystems

You can share you thought with this squad 

Backend Developer 

A real-world production incident is presented as a system design challenge: 47 orders got stuck in a 'cancelled' state with no 'created' record because SQS standard queue consumers processed events out of order within a 400ms window. The scenario involves OrderService publishing three events per order to a standard SQS queue with 5 parallel consumers feeding a Postgres-backed state machine. Four solutions are proposed: switching to SQS FIFO with MessageGroupId, adding a reorder buffer in the consumer, replacing the event stream with a Saga pattern, or making the state machine idempotent and order-agnostic. Readers are asked to pick the best solution and explain their reasoning.

7/30 Days System Design Questions!

✅ Answer: A — SQS FIFO + MessageGroupId per order_id
Why A wins:
You don’t need global ordering — you need per-order ordering. Set MessageGroupId = order_id. FIFO guarantees messages with the same group ID are delivered in exact send order, with only one consumer processing that group at a time. Your 2K orders/min still fan out across 5 workers — throughput is unchanged. Order #12345’s created → paid → cancelled always lands in sequence. Cost: a few ms latency. Ship it Monday.
<img src="https://media.daily.dev/image/upload/s--xK0Qi1wU--/f_auto/v1778657238/ugc/content_fd22d2e1-4dd8-4387-a813-0e738ed04c59?_a=BAMAMiWQ0" alt="ChatGPT Image May 12, 2026, 07_01_58 PM (1).png">

Why B is the trap (reorder buffer):
Fools senior engineers on the whiteboard. You’d end up building: in-memory buffer per order_id + TTL logic + dead-letter path for gaps + crash recovery + stuck buffer monitoring. You just rebuilt FIFO in app code. With more bugs. And you own every edge case. B is valid only when you can’t use FIFO (e.g. Kafka with a taken partition key). That’s not this situation.

Why C is wrong (Saga):
Saga is for long-running distributed transactions with compensating actions (book flight → charge card → reserve hotel). Your problem is a state stream for one entity — not a multi-service transaction. Forcing it into a Saga turns async events into sync RPC, increases coupling, and still doesn’t fix ordering.

Why D is wrong (event versioning):
Partially works — but you’d drop legitimate events. “cancelled” arrives first, moves state forward, then “created” gets rejected. DB says order is cancelled with no creation record. To make D work correctly you need full event sourcing (6-month rewrite, not a Monday fix).