One morning I opened AWS Billing and saw something that makes every DevOps engineer uncomfortable: a sharp vertical spike.
No marketing campaign.
No traffic surge.
No new feature launch.
Our AWS cost had tripled in less than 24 hours.
Here’s what happened — and the guardrail I’ll never skip again.
<hr>
<h2>The Setup: “It’s Just a Background Job”</h2>
We had a background worker running on EC2. Its job was simple:
<ul>
<li>Pull messages from SQS</li>
<li>Process media files</li>
<li>Upload results to S3</li>
<li>Delete the message</li>
</ul>
Standard stuff.
We had recently increased concurrency to “speed things up.”
Node.js worker (simplified):
<pre><code>while (true) {
 const messages = await sqs.receiveMessage({ MaxNumberOfMessages: 10 }).promise();

 await Promise.all(
 messages.map(msg =&gt; processJob(JSON.parse(msg.Body)))
 );
}
</code></pre>
It worked great in testing.
Until production.
<hr>
<h2>The Silent Problem: No Concurrency Ceiling</h2>
Under certain failure conditions:
<ul>
<li><code>processJob</code> retried internally</li>
<li>Jobs took longer</li>
<li>Messages weren’t deleted fast enough</li>
<li>More instances scaled up</li>
</ul>
Because we had:
<ul>
<li>An Auto Scaling Group</li>
<li>CPU-based scaling policy</li>
<li>No max instance cap set properly</li>
</ul>
AWS did exactly what it was told to do:
<blockquote>
“CPU high? Add instances.”
</blockquote>
It added them.
And kept adding them.
<hr>
<h2>The Cascade Effect</h2>
Here’s how it spiraled:
<ol>
<li>Jobs slow down → CPU increases</li>
<li>ASG launches more EC2 instances</li>
<li>More instances pull more SQS messages</li>
<li>Downstream dependency throttles</li>
<li>Retries increase</li>
<li>CPU spikes more</li>
<li>Repeat</li>
</ol>
It wasn’t traffic.
It was self-amplifying concurrency.
By the time we noticed:
<ul>
<li>EC2 count had doubled</li>
<li>S3 PUT requests spiked</li>
<li>Data transfer increased</li>
<li>NAT gateway costs quietly ballooned</li>
</ul>
AWS didn’t fail.
Our assumptions did.
<hr>
<h2>The Fix (What I Should Have Done First)</h2>
<h3>1️⃣ Hard Cap the Auto Scaling Group</h3>
Always define a maximum instance count.
Terraform example:
<pre><code>resource &quot;aws_autoscaling_group&quot; &quot;workers&quot; {
 max_size = 5
 min_size = 1
 desired_capacity = 2
}
</code></pre>
No infinite scaling. Ever.
<hr>
<h3>2️⃣ Limit Application-Level Concurrency</h3>
Instead of unlimited <code>Promise.all</code>, I switched to controlled parallelism:
<pre><code>import pLimit from &quot;p-limit&quot;;

const limit = pLimit(5);

await Promise.all(
 messages.map(msg =&gt;
 limit(() =&gt; processJob(JSON.parse(msg.Body)))
 )
);
</code></pre>
Infrastructure scaling + unlimited app concurrency = chaos.
Pick one.
<hr>
<h3>3️⃣ Add Circuit Breakers</h3>
If downstream dependency fails:
<ul>
<li>Stop pulling new messages</li>
<li>Pause briefly</li>
<li>Fail fast instead of retrying blindly</li>
</ul>
Backpressure is not optional in distributed systems.
<hr>
<h3>4️⃣ Billing Alerts (The Obvious One)</h3>
I added:
<ul>
<li>AWS Budget alerts at 50%, 75%, 90%</li>
<li>Cost anomaly detection</li>
<li>Slack notifications</li>
</ul>
Embarrassingly, we had none of that before.
<hr>
<h2>The Lesson Most Teams Learn Once</h2>
Cloud doesn’t break loudly.
It scales quietly.
And scaling is expensive when the trigger condition is flawed.
AWS is incredibly good at doing what you configure.
It will happily automate your mistakes.
<hr>
<h2>Conclusion / Key takeaway</h2>
Our AWS bill didn’t spike because of growth.
It spiked because we removed a limit and trusted scaling without boundaries.
In cloud systems, unbounded concurrency is a liability.
Have you ever had an AWS (or cloud) cost surprise? What guardrail did you add afterward?

Coffee Mug

The official daily.dev DevOps community. Led by thought leaders and DevOps experts. Join the Squad!

DevOps

A DevOps engineer shares a postmortem of an AWS bill tripling overnight due to a runaway Auto Scaling Group. A background SQS worker with no concurrency ceiling triggered a self-amplifying feedback loop: slow jobs raised CPU, which launched more EC2 instances, which pulled more messages, which caused more retries. The fix involved setting a hard max on the ASG in Terraform, capping application-level concurrency with p-limit, adding circuit breakers for downstream failures, and setting up AWS Budget alerts and cost anomaly detection.

Our AWS Bill Spiked 3x Overnight — It Wasn’t Traffic, It Was One Missing Limit

Did this actually happen, or is this just a figment of an AI’s imagination?

Classic positive feedback loop, autoscaling + unbounded concurrency. Cloud didn’t fail, the system lacked limits. Hard caps, backpressure, and cost alerts aren’t optional, they’re survival.