I’m working on an automated workflow for incident resolution with n8n, AI-agent and Bash scripts.
This is the visualization of the flow:
<a href="https://raw.githubusercontent.com/Bubobot-Team/automation-workflow-monitoring/refs/heads/main/assets/n8n_AI_Agent_Decision_Engine_for_Self_Healing_Server_VPS.png" target="_blank" rel="noopener nofollow">https://raw.githubusercontent.com/Bubobot-Team/automation-workflow-monitoring/refs/heads/main/assets/n8n_AI_Agent_Decision_Engine_for_Self_Healing_Server_VPS.png</a>
The components that I’m using:
<ol>
<li>
Prometheus - Grafana: For monitoring and metrics collection.
</li>
<li>
Alertmanager: To send alerts, and n8n to orchestrate a workflow
</li>
<li>
First bash script: To do a system check and analyze
</li>
<li>
First AI-agent node: To analyze the result from the system check scripts. The agent will evaluate how to interact with the issues (just notify or take immediate action).
</li>
<li>
If it’s medium and low cases, just send a notification to Discord. If it’s kind of urgent actions, go the critical flow
</li>
<li>
Second AI-agent node (in critical flow): Analyzes the actual system state and creates specific fix commands. The commands could be clean logs, try to restart some services to release the resources.
</li>
<li>
Second bash script: to run the commands under AI-agent suggestions.
</li>
<li>
Finally, send some post-mortem and reports through Discord.
</li>
</ol>
<h3>Note:</h3>
<ul>
<li>
Remember to optimize your prompt for safety validation blocks for dangerous commands in the 2nd AI-agent node (align with your stacks)
</li>
<li>
You define in the prompt what services should be prioritized, and what should not be in the 2nd AI-agent node.
</li>
</ul>
I’ve shared all materials here:
<ul>
<li>
System doctor to analyze the current state of the system: <a href="https://github.com/Bubobot-Team/sysadmin-toolkit/blob/staging/scripts/system-health/system-doctor.sh" target="_blank" rel="noopener nofollow">https://github.com/Bubobot-Team/sysadmin-toolkit/blob/staging/scripts/system-health/system-doctor.sh</a>
</li>
<li>
N8n workflow code (just copy to your n8n and run): <a href="https://github.com/Bubobot-Team/automation-workflow-monitoring/blob/main/n8n/n8n_AI_Agent_Decision_Engine_for_Self_Healing_Server_VPS.json" target="_blank" rel="noopener nofollow">https://github.com/Bubobot-Team/automation-workflow-monitoring/blob/main/n8n/n8n_AI_Agent_Decision_Engine_for_Self_Healing_Server_VPS.json</a>
</li>
<li>
Overview of the workflow: <a href="https://github.com/Bubobot-Team/automation-workflow-monitoring?tab=readme-ov-file#ai-agent-decision-engine-for-self-healing-server-vps" target="_blank" rel="noopener nofollow">https://github.com/Bubobot-Team/automation-workflow-monitoring?tab=readme-ov-file#ai-agent-decision-engine-for-self-healing-server-vps</a>
</li>
</ul>
Limitations of the AI-based decision approach (I will address this in the next part of the series):
<ul>
<li>Non-deterministic AI behavior (responses vary for identical scenarios)</li>
<li>Data privacy concerns (metrics sent to cloud APIs)</li>
<li>Consistency challenges for audit trails and debugging</li>
</ul>
I hope this helps you to apply in your server or VPS, to help you self-fix your services just in case of an emergency. I would love to hear your feedback and collaboration to improve the flow.
For a full tutorial, please at our blog post at <a href="https://bubobot.com/blog/building-an-ai-agent-decision-engine-for-self-healing-to-protect-uptime-part-1" target="_blank" rel="noopener nofollow">https://bubobot.com/blog/building-an-ai-agent-decision-engine-for-self-healing-to-protect-uptime-part-1</a>

The official daily.dev DevOps community. Led by thought leaders and DevOps experts. Join the Squad!

DevOps

A comprehensive automated workflow system that combines Prometheus monitoring, AI agents, and bash scripts to create self-healing server infrastructure. The system uses a multi-stage decision engine where AI agents analyze system health data and determine appropriate responses - from simple notifications for minor issues to automated remediation commands for critical problems. The workflow integrates with Discord for notifications and includes safety validation mechanisms to prevent dangerous command execution.

AI-Agent Decision Engine for Self-Healing Server/VPS