2 lines of code. 2 completely different storage systems. One network blip and you’re debugging missing orders in production at 2am.
You’ve just accepted an order. Now you need to:
<ol>
<li>Save it to the database</li>
<li>Publish OrderPlaced to Kafka so billing, inventory, and notifications can react</li>
</ol>
Your code looks clean:
db.save(order);
kafka.publish(“OrderPlaced”, event);
Ship it. Go home.
Three weeks later, support opens a ticket. A customer was charged. No shipment. No email. The order is in the DB, but billing never got the event.
Welcome to the dual write problem.
Those two lines are not atomic. The DB commits. The app crashes before kafka.publish() returns. Or the broker is briefly unavailable. Or the network blips for 400ms.
The order exists. The event does not. And now your microservices are permanently out of sync, and nobody knows until a customer complains.
The “obvious” fix is a distributed transaction (2PC). Don’t.
Kafka doesn’t support it. SQS and SNS don’t support it. The brokers that do support XA will stall your entire write path the moment one participant gets slow.
The industry walked away from 2PC for event-driven systems a decade ago for good reason.
Here’s what actually works: the Outbox Pattern.
Instead of publishing to the broker from your service, you insert the event into an outbox table in the exact same database transaction as your business write.
BEGIN;
INSERT INTO orders (…);
INSERT INTO outbox (event_type, payload);
COMMIT;
One transaction. Atomic. Either both rows exist, or neither do. The dual write is gone.
A separate publisher process reads the outbox and ships events to Kafka. It can be a simple poller (SELECT … FOR UPDATE SKIP LOCKED every 500ms) or Debezium tailing the Postgres WAL for sub-100ms latency.
After publishing, mark the row processed.
The tradeoff nobody mentions in the tutorials: you get at-least-once delivery, not exactly-once.
The publisher might ship an event, crash before marking it done, restart, and ship it again. Your consumers must be idempotent. Use the outbox row ID as the dedup key. This is the same tradeoff Kafka’s own “exactly-once semantics” quietly inherits.
The pattern is how Shopify moves order events across merchants, how Debezium + Postgres became the canonical production stack, and why Dapr added Outbox as a first-class building block in 2024.
If your service writes to a DB and publishes to a broker, and you’re not using Outbox, you have a bug. You just haven’t been paged for it yet.
#SystemDesign #BackendEngineering #Microservices #DistributedSystems

Joud Awad

The dual write problem occurs when saving to a database and publishing to a message broker are two separate non-atomic operations — a crash between them leaves your systems permanently out of sync. The Outbox Pattern solves this by inserting the event into an outbox table within the same database transaction as the business write, then using a separate publisher process (a poller or Debezium tailing the WAL) to relay events to Kafka. This guarantees atomicity at the write side. The key tradeoff is at-least-once delivery, not exactly-once, so consumers must be idempotent using the outbox row ID as a dedup key. Distributed transactions (2PC) are not a viable alternative for event-driven systems.