This blog comes from Coroot contributor Arie van den Heuvel: engineer, a System and Application Management Specialist, and a valued member of our open source community. You can read more of Arie’s writing and support the resource articles he has contributed to open source <a href="https://solipsistic-sysadmin.blogspot.com/" target="_blank" rel="noopener nofollow">on his blog.</a>
<img src="https://media.daily.dev/image/upload/s--mvl5spCd--/f_auto/v1764961373/ugc/content_ca86953a-92d7-4583-9dc5-55f76617642a?_a=BAMAK+ZW0" alt="Arie">
When running a home server consisting of one or more nodes with some or all services in Docker, you may find yourself wanting to monitor your environment. Or even better, attain full observability.
The frequent recommendation for this is a combination of Prometheus with Grafana. But this solution requires a lot of work to fully configure, in addition to work on one’s applications and setup for full visibility. Another possibility is to use the free tier of NewRelic, which has the advantage of remote insights on metrics and logs. Again, this requires additional work on containers or applications to have a more refined visibility of your services.
For those not running Linux, an honourable mention to use as a solution would be <a href="https://github.com/henrygd/beszel" target="_blank" rel="noopener nofollow">Beszel.</a> Beszel can be run as a local service or in docker. It consists of a web front-end and an agent that can be used on multiple systems that run Windows and MacOS. Installation is an easy job in docker. Once it’s running, Beszel provides insightful information with system metrics, docker services, and even logs.
My personal choice for monitoring a home server system is <a href="https://github.com/coroot/coroot" target="_blank" rel="noopener nofollow">Coroot. </a> In the following blog, I’ll detail how I used Coroot to set up observability for my homelab, which you can then adopt for your own setup.
<h1>Observability with Coroot</h1>
In my current setup on a <a href="https://rockylinux.org/" target="_blank" rel="noopener nofollow">Rocky Linux</a> 9.x system, Coroot runs on a <a href="https://github.com/ClickHouse/ClickHouse" target="_blank" rel="noopener nofollow">Clickhouse</a> server to store metrics, logs, traces and profiles, in addition to the <a href="https://github.com/coroot/coroot-node-agent" target="_blank" rel="noopener nofollow">Coroot node-agent </a> and <a href="https://github.com/coroot/coroot-cluster-agent" target="_blank" rel="noopener nofollow">Coroot cluster-agent.</a> The Coroot node-agent automatically collects all service metrics and logs using eBPF, while the cluster-agent provides detailed information on databases like MySQL, Postgres or Redis.
<img src="https://media.daily.dev/image/upload/s--BXOK5SK_--/f_auto/v1764962316/ugc/content_2ed258fd-0f34-4b7a-b939-9e206fa6548c?_a=BAMAK+ZW0" alt="AIRCA__">
Another advantage Coroot presents is the use of <a href="https://coroot.com/ai" target="_blank" rel="noopener nofollow">AI-powered Root Cause Analysis</a>, which provides instantaneous and helpful insights for investigating incidents. With a <a href="https://docs.coroot.com/ai/coroot-cloud/" target="_blank" rel="noopener nofollow">Coroot Cloud account</a>, you will have ten helpful analyses for free each month. Even without AI, the data presented with Coroot with standard alerts based on best metric practices is pretty insightful and helps to make your setup even better.
Coroot services run in docker through a docker-compose file. In a normal Coroot setup Prometheus is used, but in this setup I have configured Clickhouse, <a href="%5Burl%5D(https://docs.coroot.com/configuration/clickhouse/)" target="_blank" rel="noopener nofollow">which is a supported alternative.
</a>
<h1>Clickhouse as a Local Service</h1>
I have Clickhouse running as a local service. This setup allows for better control and convenience when scaling down memory usage of Clickhouse, scaling down logging on disk and the database, and simplifies making changes to the data. The only downside to note is this setup requires updating Clickhouse manually with <code>yum/dnf.</code>
<h1>Installing Clickhouse</h1>
Installing Clickhouse is easily achieved by adding the repo, installing Clickhouse, and making a few quick adjustments before starting it up.
<pre><code>
sudo dnf install -y yum-utils
sudo dnf-config-manager --add-repo
https://packages.clickhouse.com/rpm/clickhouse.repo
sudo dnf install -y clickhouse-server clickhouse-client

</code></pre>
Before starting the service create file <code>/etc/clickhouse-server/config.d/z_log_disable.xml</code> and insert the following content in the file:
<pre><code>
​&lt;?xml version=&quot;1.0&quot;?&gt;
&lt;clickhouse&gt;
&lt;asynchronous_metric_log remove=&quot;1&quot;/&gt;
&lt;metric_log remove=&quot;1&quot;/&gt;
&lt;latency_log remove=&quot;1&quot;/&gt;
&lt;query_thread_log remove=&quot;1&quot; /&gt;
&lt;query_log remove=&quot;1&quot; /&gt;
&lt;query_views_log remove=&quot;1&quot; /&gt;
&lt;part_log remove=&quot;1&quot;/&gt;
&lt;session_log remove=&quot;1&quot;/&gt;
&lt;text_log remove=&quot;1&quot; /&gt;
&lt;trace_log remove=&quot;1&quot;/&gt;
&lt;crash_log remove=&quot;1&quot;/&gt;
&lt;opentelemetry_span_log remove=&quot;1&quot;/&gt;
&lt;zookeeper_log remove=&quot;1&quot;/&gt;
&lt;/clickhouse&gt;

</code></pre>
After this adjust cache sizes in <code>/etc/clickhouse-server/config.xml:</code>
<pre><code>&lt;mark_cache_size&gt;268435456&lt;/mark_cache_size&gt;
&lt;index_mark_cache_size&gt;67108864&lt;/index_mark_cache_size&gt;
&lt;uncompressed_cache_size&gt;16777216&lt;/uncompressed_cache_size&gt;

</code></pre>
Adjust memory usage ratio in <code>/etc/clickhouse-server/config.xml:</code>
<pre><code>
&lt;max_server_memory_usage_to_ram_ratio&gt;0.75&lt;/max_server_memory_usage_to_ram_ratio&gt;

</code></pre>
Lower the tread pool size in <code>/etc/clickhouse-server/config.xml:</code>
<pre><code>&lt;!-- max_thread_pool_size&gt;10000&lt;/max_thread_pool_size&gt; --&gt;

</code></pre>
And starting things up:
<pre><code>
sudo systemctl deamon-reload
sudo systemctl enable clickhouse-server
sudo systemctl start clickhouse-server

</code></pre>
<h1>Installing Coroot</h1>
First, check if your Linux system is using kernel 5.1 or later (although 4.2 is also supported.) This installation is different from the <a href="https://docs.coroot.com/?env=docker" target="_blank" rel="noopener nofollow">original docker-compose file.</a>
Prometheus is not used in this setup, and Clickhouse runs as a local service. Another distinction is the retention of the data, which is normally set to seven days for traces, logs, profiles and metrics. Coroot also typically stores its own local cache for metrics for 30 days.
In this setup, the data retention stored in Clickhouse is set up for 14 days. With eighteen local and docker services, the amount of data kept for all of this is 3GB on average in my system.
Coroot, its node-agent, and cluster-agent, run as a docker service with a docker-compose that you create locally. This is achieved by inserting the following content in a locally created <code>docker-compose.yaml:</code>
<pre><code>
name: coroot

volumes:
node_agent_data: {}
cluster_agent_data: {}
coroot_data: {}

services:
coroot:
restart: always
image: ghcr.io/coroot/coroot${LICENSE_KEY:+-ee} # set &#x27;coroot-ee&#x27; as the image if LICENSE_KEY is defined
pull_policy: always
user: root
volumes:
- coroot_data:/data
ports:
- 8080:8080
command:
- &#x27;--data-dir=/data&#x27;
- &#x27;--bootstrap-refresh-interval=15s&#x27;
- &#x27;--bootstrap-clickhouse-address=127.0.0.1:9000&#x27;
- &#x27;--bootstrap-prometheus-url=http://127.0.0.1:9090&#x27;
- &#x27;--global-prometheus-use-clickhouse&#x27;
- &#x27;--global-prometheus-url=http://127.0.0.1:9090&#x27;
- &#x27;--global-refresh-interval=15s&#x27;
- &#x27;--cache-ttl=31d&#x27;
- &#x27;--traces-ttl=21d&#x27;
- &#x27;--logs-ttl=21d&#x27;
- &#x27;--profiles-ttl=21d&#x27;
- &#x27;--metrics-ttl=21d&#x27;
environment:
- LICENSE_KEY=${LICENSE_KEY:-}
- GLOBAL_PROMETHEUS_USE_CLICKHOUSE
- CLICKHOUSE_SPACE_MANAGER_USAGE_THRESHOLD=75 # Set cleanup threshold to 75%
- CLICKHOUSE_SPACE_MANAGER_MIN_PARTITIONS=2 # Always keep at least 2 partitions
network_mode: host

node-agent:
restart: always
image: ghcr.io/coroot/coroot-node-agent
pull_policy: always
privileged: true
pid: &quot;host&quot;
volumes:
- /sys/kernel/tracing:/sys/kernel/tracing
- /sys/kernel/debug:/sys/kernel/debug
- /sys/fs/cgroup:/host/sys/fs/cgroup
- node_agent_data:/data
command:
- &#x27;--collector-endpoint=http://192.168.1.160:8080&#x27;
- &#x27;--cgroupfs-root=/host/sys/fs/cgroup&#x27;
- &#x27;--wal-dir=/data&#x27;

cluster-agent:
restart: always
image: ghcr.io/coroot/coroot-cluster-agent
pull_policy: always
volumes:
- cluster_agent_data:/data
command:
- &#x27;--coroot-url=http://192.168.1.160:8080&#x27;
- &#x27;--metrics-scrape-interval=15s&#x27;
- &#x27;--metrics-wal-dir=/data&#x27;
depends_on:
- coroot


</code></pre>
After creating this file and making any adjustments to your own likings and network preferences, type docker compose up -d and go to your IP address on port 8080. Here you have access to Coroot, and are now prompted to give admin credentials!
In my setup, <a href="https://github.com/containrrr/watchtower" target="_blank" rel="noopener nofollow">Watchtower</a> takes care of updating docker containers, which works great with Coroot.
As a final sidenote: there are already some helpful hints and pointers present within Coroot for setting things up. In my case, there was information available that helped observe a Postgres database. Don’t forget to use the given commands as the admin/postgres user to make it work.
Happy homelab observing! 🐧

Star

Self-hosted refers to the practice of running your own applications and services on your own servers or devices, giving you full control over your data and how it is managed.

selfhosted

A detailed guide for setting up observability in a Docker-based homelab using Coroot, an open-source monitoring solution. The tutorial covers installing Clickhouse as a local service for storing metrics, logs, traces, and profiles, then configuring Coroot with its node-agent and cluster-agent via Docker Compose. The setup uses eBPF for automatic metric collection, includes AI-powered root cause analysis, and provides configuration optimizations for memory usage and data retention (14 days). The guide offers an alternative to traditional Prometheus/Grafana stacks with less configuration overhead.

Monitoring a Docker Homelab with Open Source

learnt a lot. thanks a ton for sharing