Docker Model Runner now integrates vLLM inference engine and safetensors models, enabling high-throughput AI inference on NVIDIA GPUs. The integration automatically routes requests between llama.cpp and vLLM based on model format (GGUF vs safetensors), allowing developers to prototype locally and scale to production using the same Docker commands. Currently available for x86_64 with NVIDIA GPUs, with WSL2/Docker Desktop and DGX compatibility planned.
Table of contents
Expanding Docker Model Runner’s CapabilitiesWhy vLLM?How vLLM WorksWhy Multiple Inference Engines?Safetensors (vLLM) vs. GGUF (llama.cpp): Choosing the Right FormatvLLM-compatible models on Docker HubAvailable Now: x86_64 with NvidiaWhat’s Next?Sort: