A practical guide to the LLM router pattern for production AI apps, covering how to route requests to different models based on task complexity, implement fallbacks across providers, and cut costs without quality regressions. The author shares a personal experience of reducing an AI bill by 70% by routing easy requests to cheaper models. The post covers three routing layers (provider routing, model routing within a provider, and strategy routing across model families), compares tools like Vercel AI Gateway, OpenRouter, Portkey, and LiteLLM, and provides a bucketing framework for classifying requests. It also covers fallback best practices, observability requirements, anti-patterns to avoid, and a step-by-step migration path from a single-model app.
Table of contents
Why One Model Is The Wrong DefaultWhat An LLM Router Actually IsThe Tools That Make This Manageable In 2026How To Decide Which Model Handles A RequestFallbacks That Hold UpCost Control Without Quality RegressionObservability For The Router ItselfPatterns That Almost Always BackfireWhat To Build FirstWhere This Is GoingSort: