72 Techniques to Optimize LLMs in Production

A comprehensive breakdown of 72 techniques for optimizing LLMs in production, organized into nine layers: model compression (quantization, distillation, pruning), attention and architecture (FlashAttention, PagedAttention, MQA/GQA/MLA), decoding strategies (speculative decoding, Medusa, EAGLE), KV cache management (prefix caching, token eviction, quantization), batching and scheduling (continuous batching, prefill-decode disaggregation), parallelism and kernels (tensor/pipeline/expert parallelism, CUDA graphs), application caching (semantic caching, embedding deflection), input/output shaping (prompt compression, context pruning), and routing/cost strategies (model cascading, classifier routing). The post explains how stacking these techniques can achieve a 5-8x cost-per-token improvement over naive FP16 deployments, noting that LLM inference prices have dropped roughly 10x per year largely due to serving stack improvements.