<h2>What happened</h2>
<p>At KubeCon + CloudNativeCon Europe 2026, Red Hat, IBM Research, and Google Cloud donated llm-d to the CNCF as a Sandbox project. Originally launched in May 2025 by Red Hat, Google Cloud, IBM Research, CoreWeave, and NVIDIA, llm-d is an open-source framework for running LLM inference as a distributed, production-grade workload on Kubernetes.</p>
<p>The donation came alongside a broader CNCF push to make Kubernetes a serious platform for AI inference. The foundation published Kubernetes AI Requirements (KARs) v1.35, which now mandates stable in-place pod resizing and workload-aware scheduling. The KAR program currently counts 31 certified platforms, and a “Verify Conformance Bot” for third-party validation is in the works.</p>
<h2>What llm-d actually does</h2>
<p>The project targets a specific gap: the space between high-level control planes like KServe and low-level inference engines like vLLM. Running a large model on a single GPU node works fine at small scale, but production inference needs to spread across multiple accelerators and adapt to shifting load — and vanilla Kubernetes wasn’t built for that.</p>
<p>llm-d addresses this with a few concrete capabilities:</p>
<ul>
<li><strong>Prefill/decode disaggregation</strong> — the two phases of LLM inference run on separate nodes and scale independently, so a spike in new requests doesn’t starve ongoing generation</li>
<li><strong>KV-cache-aware routing</strong> — traffic is directed based on what’s already cached, reducing redundant computation</li>
<li><strong>Inference-aware autoscaling</strong> — the scheduler accounts for both traffic patterns and hardware characteristics, not just CPU/memory</li>
<li><strong>Hardware-agnostic serving</strong> — works across NVIDIA, AMD, Intel, and Google TPUs</li>
</ul>
<p>It builds on vLLM and integrates with the Kubernetes Gateway API Inference Extension and LeaderWorkerSet.</p>
<h2>Early numbers</h2>
<p>Benchmarks from the v0.5 release on Qwen3-32B show near-zero time-to-first-token and roughly 120,000 tokens per second throughput, compared to rapid degradation under load with a standard Kubernetes service. Google Cloud’s internal testing reported 2x improvements in time-to-first-token.</p>
<p>The project also aims to establish open, reproducible inference benchmarks as a neutral industry standard — something the ecosystem currently lacks.</p>
<h2>Who’s involved</h2>
<p>Founding collaborators beyond the three donors include NVIDIA, CoreWeave, AMD, Cisco, Hugging Face, Intel, Lambda, and Mistral AI.</p>
<h2>What’s next</h2>
<p>The CNCF is also tracking related ecosystem work: AWS’s Kube Resource Orchestrator (KRO), the Kueue job queueing system, and NVIDIA’s open-source AI Cluster Runtime (AICR). Future KAR standards for Sovereign AI are planned, and llm-d’s own roadmap targets multi-modal workloads, multi-LoRA optimization, and deeper vLLM integration.</p>
<p>CNCF executive director Jonathan Bryce has been direct about the direction: Kubernetes should become the dominant platform for AI inference, including edge deployments running smaller models.</p>


Collections

Red Hat, IBM Research, and Google Cloud have donated llm-d to the CNCF as a Sandbox project, positioning it as a Kubernetes-native framework for distributed LLM inference at production scale. llm-d fills the gap between high-level control planes like KServe and low-level engines like vLLM by providing prefill/decode disaggregation, KV-cache-aware routing, inference-aware autoscaling, and hardware-agnostic serving across NVIDIA, AMD, Intel, and Google TPUs. Early benchmarks on Qwen3-32B show ~120,000 tokens/second throughput with near-zero time-to-first-token, and Google Cloud reported 2x improvements in time-to-first-token versus standard Kubernetes services. The CNCF also published Kubernetes AI Requirements (KARs) v1.35 mandating stable in-place pod resizing and workload-aware scheduling, with 31 certified platforms currently. The project's roadmap includes multi-modal workloads, multi-LoRA optimization, and deeper vLLM integration.

llm-d joins the CNCF as a Kubernetes-native framework for distributed LLM inference