A repeatable framework for comparing LLM outputs in production is essential as prompts, models, and routing logic evolve. Manual review and ad-hoc prompting are insufficient at scale due to inconsistency and lack of baselines. Effective evaluation requires tracking metrics across three dimensions: hallucination detection, relevance/correctness, and safety/quality. Metrics can be deterministic (regex, JSON validation) or model-based (LLM-as-a-judge), applied at span, trace, or session granularity. Arize treats evaluation as a continuous operational loop attached to traces, with pre-built and custom evaluators that produce explainable scores. Portkey's AI Gateway complements this by providing routing and orchestration so teams can compare models and prompt versions under identical conditions, connecting evaluation results back to specific routing decisions for actionable iteration.
Table of contents
Is manual review and one-off prompting enough to compare LLM outputs?Metrics to consider when comparing LLM outputsHow Arize approaches evalsCompleting the loop with Portkey: routing and orchestration for evaluationsSort: