Running LLMs on Kubernetes requires a different security approach. Learn how to build an LLM gateway that addresses OWASP Top 10 risks like prompt injection and data leakage.

MetalBear

Running LLMs on Kubernetes requires application-layer security controls beyond what Kubernetes provides. The OWASP LLM Top 10 identifies risks like prompt injection, sensitive data leakage, supply chain vulnerabilities, and excessive tool permissions. An LLM gateway acts as a policy enforcement layer, validating inputs, filtering outputs, restricting model access, and controlling tool permissions. The article demonstrates building a reference gateway implementation, using mirrord for fast local development against cluster resources, and Cloudsmith for model artifact governance with versioning and access controls.

LLMs on Kubernetes: Same Cluster, Different Threat Model

Understanding what you’re actually running #

The challenges of running LLMs in Kubernetes #

Development: fast iteration with mirrord #

Production: supply chain governance with Cloudsmith #