Let’s say you’ve got an LLM running on Kubernetes. Pods are healthy, logs are clean, users are chatting. Everything looks fine. But here’s the thing: Kubernetes…

CNCF's platform is a leading organization driving cloud-native technologies and standards, offering insights into container orchestration, microservices architecture, and cloud-native infrastructure. Through whitepapers, case studies, and community events, CNCF provides insights into adopting cloud-native practices and technologies. Developers and DevOps teams can learn about Kubernetes, Prometheus, and other CNCF projects to build and operate scalable and resilient cloud-native applications.

CNCF

Running LLMs on Kubernetes introduces a distinct threat model that Kubernetes itself cannot address. While the infrastructure handles scheduling and isolation, it has no awareness of what the workloads do with untrusted input. Four OWASP LLM Top 10 risks are particularly relevant for Kubernetes operators: prompt injection (LLM01), sensitive information disclosure (LLM02), supply chain risks (LLM03), and excessive agency (LLM06). Each maps to familiar infrastructure security patterns — input validation, output filtering, image provenance, and least-privilege access — but applied to probabilistic, language-based systems. The recommended approach is a dedicated policy layer (an LLM-aware API gateway) sitting in front of the model runtime, keeping inference and policy concerns separate. Tools like LiteLLM, Kong AI Gateway, Portkey, and kgateway are highlighted as options. A follow-up post will cover a reference implementation.

LLMs on Kubernetes Part 1: Understanding the threat model

Understanding what you’re actually running

OWASP LLM Top 10: A framework for understanding risks

Four risks that Kubernetes operators need to understand