Understanding disaggregated GenAI model serving with llm-d

llm-d is an open source, cloud-native system for high-scale LLM deployments that addresses latency and memory-bandwidth bottlenecks by disaggregating the inference workload across dedicated hardware. It splits processing into four Kubernetes-managed components: an inference scheduler (adaptive load balancer using Prometheus metrics), a KV cache manager, a prefill worker for compute-intensive prompt processing, and a decode worker for token generation. The post explains why organizations with data sovereignty requirements would self-host LLMs, highlights the availability of competitive open-weight models, and shares experimental Juju charms the author built to simplify llm-d deployments on Ubuntu without requiring deep Kubernetes expertise.