<p>Hey guys,</p>
<p>I’m working on a monitoring setup that automates basic incident resolutions.</p>
<p>I’m using <strong>Prometheus</strong> - <strong>Grafana</strong> for monitoring, <strong>Alertmanager</strong> to send alerts, and <strong>n8n</strong> to orchestrate a workflow, then an <strong>AWS Lambda</strong> function to restart the services. “Restart services” is a kind of demo action, you can customize it for your needs.</p>
<h2>Components to setup?</h2>
<h3>1. Prometheus:</h3>
<p>I configure some basic rules to alert when CPU/Memory exceeds a threshold. When the thresholds are exceeded, it will send a webhook to n8n system.</p>
<h3>2. N8n flow:</h3>
<p>Get information, analyze the metrics, calculate the business hours or incident duration, and send alerts to Discord or escalate to PagerDuty.</p>
<h3>3. AI agent (in n8n):</h3>
<p>I define a prompt to check for the input. I will consider the metrics and current contexts to decide whether to restart the services or not.</p>
<h3>4. Lambda function:</h3>
<p>Receive the commands from AI agent and process if necessary. Currently, I grant it to restart an EC2 instance to make the service available again when the system overloaded.</p>
<p>I hope this helps you to apply an automated stack in your team. I’ve shared the example materials in those repositories:</p>
<ul>
<li>
<h3>One-click to set up Prometheus - Alert Manager - Grafana at</h3>
</li>
</ul>
<p><a href="https://github.com/Bubobot-Team/monitoring-stack/tree/main/stacks/prometheus-stack" target="_blank" rel="noopener nofollow">https://github.com/Bubobot-Team/monitoring-stack/tree/main/stacks/prometheus-stack</a></p>
<ul>
<li>
<h3>N8n workflow in JSON format (just copy into your n8n dashboard):</h3>
</li>
</ul>
<p><a href="https://github.com/Bubobot-Team/automation-workflow-monitoring" target="_blank" rel="noopener nofollow">https://github.com/Bubobot-Team/automation-workflow-monitoring</a></p>
<p>Btw, just wondering, what recovery actions would you automate? (e.g., disk cleanup, rollback deployments). I would like to hear your feedback to improve the current flow.</p>
<p>For more detailed steps, visit my blog post here: <a href="https://bubobot.com/blog/automated-incident-response-workflows-with-n8n-and-monitoring-tools" target="_blank" rel="noopener nofollow">https://bubobot.com/blog/automated-incident-response-workflows-with-n8n-and-monitoring-tools</a></p>

The official daily.dev DevOps community. Led by thought leaders and DevOps experts. Join the Squad!

DevOps

Learn how to build an automated incident response workflow using Prometheus for monitoring, n8n for orchestration, and AWS Lambda for executing custom actions. This setup includes sending alerts using Alertmanager and possible integrations with Discord and PagerDuty.

Build an incident response workflow with n8n + Prometheus

<p>Would love to hear your feedbacks then I can improve my workflow!</p>
