Why Your AI Search Evaluation Is Probably Wrong (And How to Fix It)

Ad-hoc AI search evaluation leads to costly infrastructure mistakes. A five-step framework helps build rigorous, reproducible benchmarks: (1) define domain-specific quality criteria tied to business impact, (2) build a golden test set of 100-200 queries with a graded rubric and inter-annotator agreement measured via Cohen's Kappa, (3) run controlled parallel comparisons with multiple trials per query to account for stochasticity, (4) use LLM judges calibrated against human raters, and (5) measure evaluation stability using the Intraclass Correlation Coefficient (ICC) to distinguish genuine capability differences from random noise. ICC is highlighted as critical—two providers with identical accuracy can have vastly different reliability profiles, and ignoring this leads to deploying unpredictable systems in production.