A practical framework for offline evaluation of production LLM agents, structured around three pillars: routing evaluation, LLM-as-judge assessment, and RAG evaluation. Routing evaluation catches over- and under-routing failures using deterministic and LLM-based approaches. LLM-as-judge covers factual accuracy, reasoning
Table of contents
Introduction & ContextThe System under EvaluationThree Pillars of Offline EvaluationImplementation & IntegrationConclusionSort: