How I doubled my GPU efficiency without buying a single new card

A capacity planning engineer discovered that LLM inference is fundamentally two distinct workloads — prefill (prompt processing) and decode (token generation) — running on the same hardware, causing massive GPU underutilization. Prefill saturates compute at 90–95% for ~200ms, while decode runs at only 20–30% utilization for several seconds due to memory bandwidth constraints. By splitting a 48-GPU H100 cluster into dedicated prefill and decode pools (disaggregated inference), compute efficiency roughly doubled with no new hardware. The decode pool improved from 30% to 70%+ bandwidth utilization through better batching, P99 inter-token latency flattened, and the customer saved $600–800K annually on a $2M GPU bill. Tools like vLLM, SGLang, NVIDIA Dynamo, and the open-source llm-d project now support this architecture natively, and it's already in production at Perplexity, Meta, LinkedIn, and Mistral.