Duplicates aren’t an edge case you might encounter — they’re a guaranteed property of the platform. Your architecture must account for them.
SQS retried the message. Lambda re-invoked. No bug — just a system behaving as designed without idempotency.
This is the part every tutorial skips: duplicates don’t come from one place. They come from everywhere.
→ SQS: visibility timeout expires before processing finishes. Message becomes visible again.
→ API Gateway: client taps “Pay” and the network drops after your backend succeeds. Client retries.
→ EventBridge: target invocation fails mid-flight. Automatic retry fires.
→ Lambda async: built-in retry policy sends the same event up to 3 times by default.
AWS Lambda Powertools solves this with a single decorator.
The idempotency utility uses DynamoDB as a persistence layer. When an event arrives, it extracts an idempotency key, checks DynamoDB with a conditional write, and either processes the event or returns the cached result.
THE FLOW
1️⃣ Event arrives. Extract key via JMESPath expression.
2️⃣ Key not found. Acquire IN_PROGRESS lock, process, store result as COMPLETED.
3️⃣ Key found + COMPLETED. Return cached response. No re-processing.
4️⃣ Key found + IN_PROGRESS. Block. Concurrent duplicate detected.
5️⃣ Same key, different payload. Reject. Payload mismatch caught.
One decorator. Five lines of config. Zero double-charges.
<pre><code>from aws_lambda_powertools.utilities.idempotency import (
 DynamoDBPersistenceLayer, IdempotencyConfig, idempotent
)

persistence = DynamoDBPersistenceLayer(table_name=&quot;idempotency-store&quot;)

config = IdempotencyConfig(
 event_key_jmespath=&quot;payment_id&quot;,
 expires_after_seconds=3600,
 use_local_cache=True,
)

@idempotent(config=config, persistence_store=persistence)
def handler(event, context):
 return process_payment(event[&quot;payment_id&quot;], event[&quot;amount&quot;])
</code></pre>
The DynamoDB table is PAY_PER_REQUEST with TTL enabled. You get near-zero cost and automatic cleanup. Plus, one table serves all your Lambda functions.
COMMON MISTAKES WITH IDEMPOTENCY KEYS
❌ Using Lambda’s requestId. It changes every invocation.
❌ Using SQS receiptHandle. It changes on every delivery.
❌ Hashing the full payload without normalization. Field ordering breaks the hash.
THE RIGHT APPROACH
Business identity + operation intent. For payments, that means merchant_id + client_request_id. For order processing, it is tenant_id + cart_checkout_id.
Set your TTL to match your retry window.
<ul>
<li>Payments? 24 hours to 7 days.</li>
<li>Webhook ingestion? 1 to 7 days.</li>
</ul>
If your Lambda function modifies a database or triggers an external action, it needs to be idempotent. Powertools makes it a one-line change instead of a weekend project.
#AWS #ServerlessArchitecture #SystemDesign #DistributedSystems #SoftwareEngineering

Joud Awad

Duplicates are a guaranteed property of distributed systems on AWS — SQS retries, API Gateway client retries, EventBridge retries, and Lambda async retries all produce them. AWS Lambda Powertools' idempotency utility solves this using DynamoDB as a persistence layer with a simple Python decorator. The flow extracts an idempotency key via JMESPath, uses conditional DynamoDB writes to track IN_PROGRESS and COMPLETED states, and returns cached results for duplicate events. Common mistakes include using Lambda's requestId or SQS receiptHandle as keys. The right approach is combining business identity fields like merchant_id and client_request_id, with TTL set to match the retry window (24 hours to 7 days for payments).