GPUs are everywhere powering LLM inference, model training, video processing, and more. Kubernetes is often where these workloads run. But using GPUs in Kubernetes isn’t as simple as using CPUs.
You need the right setup. You need efficient scheduling. And most importantly you need visibility.
This post walks through how to run GPU workloads on Kubernetes, how to virtualize them efficiently, and how to use open source to monitor everything with zero instrumentation.
<h1>Running GPU Workloads on Kubernetes</h1>
Running GPU workloads on Kubernetes is totally doable. But it takes a bit of setup.
It starts with your nodes. Whether you’re running in the cloud or on bare metal, your cluster needs machines with physical GPUs. Most cloud providers support GPU-enabled node pools, and provisioning them is usually straightforward.
Once the hardware is in place, the next step is software. For Kubernetes to schedule and run GPU workloads, it needs:
<ul>
<li>NVIDIA GPU drivers, installed on each node</li>
<li>The NVIDIA container runtime, so containers can access the GPU</li>
<li>The NVIDIA device plugin, so Kubernetes knows how to handle GPU resource requests</li>
</ul>
You can install all of this manually. But it’s fiddly and error-prone. That’s where the NVIDIA GPU Operator comes in. It automates the whole setup: installing drivers, configuring the runtime, and deploying the device plugin. Once that’s done, your cluster is GPU-ready.
After that, requesting a GPU is simple. Just add this to your pod spec:
<pre><code>resources:
 limits:
 nvidia.com/gpu: 1
</code></pre>
Kubernetes will handle the rest: scheduling your pod onto a node with an available GPU and assigning it to the container.
Of course, not every workload needs an entire GPU to itself. And that’s where GPU virtualization becomes really useful.
<h1>Virtualizing GPUs in Kubernetes</h1>
By default, Kubernetes treats GPUs as exclusive resources. One pod per device. But in many real-world cases, that’s overkill. The GPU Operator supports two forms of GPU virtualization that let you safely share a GPU between workloads:
Time-Slicing: Multiple containers take turns using the GPU in rapid bursts. It’s a great fit for bursty inference workloads, batch jobs, or anything that doesn’t require ultra-low latency.
MIG (Multi-Instance GPU): Available on GPUs like the A100 and H100, MIG lets you partition a single physical GPU into several hardware-isolated instances. Each one behaves like its own dedicated GPU, with its own memory, cache, and compute cores.
Virtualization makes GPUs way more flexible:
<ul>
<li>You stop wasting an entire GPU on a tiny batch job</li>
<li>You get much better overall utilization</li>
<li>You can safely share GPUs across apps without them stepping on each other</li>
</ul>
And you finally have a shot at balancing cost and performance. It’s a game-changer, but only if you can actually see what’s going on.
<h1>What observability looks like once GPUs are in play</h1>
So, the cluster is set up, the workloads are running, and maybe you’ve even started virtualizing GPUs to get better efficiency. Now comes the tricky part – actually understanding what’s happening.
From the infrastructure side, we want to know:
<ul>
<li>How many GPU-enabled nodes do we have right now?</li>
<li>Which GPUs are actually doing work, and which are just burning budget?</li>
<li>What’s the current GPU and memory utilization across the fleet?</li>
</ul>
And sure, if you’re in the cloud, temperature and power draw might feel like someone else’s problem. But it’s still good to know. Somewhere out there, your model is warming the planet one token at a time. Mother Nature says hi. 🌱
From the application side, the questions change:
<ul>
<li>Which GPUs is this app actually using?</li>
<li>How much compute and memory is it consuming?</li>
<li>Is it sharing the GPU with something else?</li>
<li>And if so who’s the noisy neighbor hogging all the resources?</li>
</ul>
This isn’t just about curiosity. It’s about avoiding slowdowns, catching inefficiencies, and making smart scaling decisions. But here’s the catch: Kubernetes doesn’t tell you any of this.
<h1>Out-of-the-box GPU observability</h1>
<a href="https://github.com/coroot/coroot" target="_blank" rel="noopener nofollow">Coroot</a> is an open source tool that uses eBPF to make any GPU-powered system observable with zero-configuration. It talks directly to the GPU using NVIDIA’s NVML library the same one behind nvidia-smi. This way you can see what’s happening on your GPUs with no guesswork.
On startup, the agent looks for <a href="http://libnvidia-ml.so" target="_blank" rel="noopener nofollow">libnvidia-ml.so</a> in all the usual (and unusual) places whether it’s installed by the GPU Operator, a package manager, or manually dropped in. If it finds the library, it loads it and starts gathering data.
From there, it:
<ul>
<li>Discovers all available GPUs on the node</li>
<li>Collects real-time metrics utilization, memory usage, temperature, power draw</li>
<li>Tracks per-process usage using NVML’s process telemetry</li>
<li>Maps each process back to its container and pod, using Coroot’s existing PID-to-container tracking</li>
</ul>
So instead of “PID 12345 is using GPU 0,” you get “this container in this pod is using 78% of GPU-xxxx”
When it comes to virtualized GPUs, Coroot sees which containers are tied to which GPU UUIDs, even when multiple workloads are time-sharing or using MIG slices on the same physical device. That means:
<ul>
<li>You can see which apps are sharing the same GPU</li>
<li>Understand how each one is using it</li>
<li>Spot noisy neighbors</li>
</ul>
All of this is automatic. Just install the agent and let Coroot do the rest.
Once the Coroot agent discovers the GPUs and starts collecting data, all of it flows straight into the UI ready to explore without any dashboards to build or metrics to stitch together.
Let’s walk through how this looks in practice.
<h1>Node-level GPU overview</h1>
<img src="https://media.daily.dev/image/upload/s--U2cVrw_U--/f_auto/v1768930531/ugc/content_abfebbd6-3d04-496d-bcce-ab721f3c58a0?_a=BAMAMiiu0" alt="Screenshot-2025-06-04-at-19.16.27-1024x572 (1)">
On the node view, Coroot shows everything you’d want to know about the GPUs attached to a specific machine:
<ul>
<li>GPU utilization over time</li>
<li>GPU memory usage</li>
<li>Top consumers of both compute and memory</li>
<li>Temperature and power draw</li>
</ul>
This isn’t just “Hey, GPU usage went up.” You can see which containers are contributing to that load and whether the same GPU is being shared between apps.
<h1>App-level GPU breakdown</h1>
<img src="https://media.daily.dev/image/upload/s--RZsTZfdW--/f_auto/v1768930553/ugc/content_5814ab9f-411c-4631-8d9a-e7cd4f4eb3d2?_a=BAMAMiiu0" alt="screencapture-demo-coroot-p-tbuzvelk-applications-default-Deployment-ollama-GPU-2025-06-04-19_17_18-1024x862 (1)">
This is where things get real. If your app is slow, you can check GPU usage alongside CPU, memory, logs, traces, and everything else in one place.
<ul>
<li>How much GPU time your app is using</li>
<li>How much memory it’s burning</li>
<li>Which containers are sharing the GPU, and how much each one is using</li>
</ul>
<h1>Conclusion</h1>
GPU workloads aren’t cheap. And they’re rarely simple. Whether you’re running a single inference service or managing dozens of shared GPUs, you need real visibility.
Observability is the same: not cheap, rarely simple. But with open source tools, managing your infra can be accessible to everyone - not just massive companies, but small businesses, startups, and homelab projects everywhere across the globe. <a href="https://github.com/coroot/coroot" target="_blank" rel="noopener nofollow">Give it a try</a> on your system next, and share feedback to help make good observability available for everyone.

The squad for Kubernetes afficionados. 
All things Kubernetes: best practices, tools, tricks, new features, operator building. 
For experienced Kubernetes administrators and Kube-newbies alike.


KubeSquad

GPUs in Kubernetes require proper setup including NVIDIA drivers, container runtime, and device plugins, which can be automated using the NVIDIA GPU Operator. GPU virtualization through time-slicing and MIG enables efficient resource sharing across workloads. Coroot is an open source eBPF-based tool that provides zero-configuration GPU observability by using NVIDIA's NVML library to track real-time metrics like utilization, memory usage, temperature, and power draw at both node and application levels, including per-container visibility for virtualized GPU workloads.

How To Make GPUs on Kubernetes Observable