To evaluate search we typically build a judgment list We transform clickstream data into evaluation data This labels a result...

Software Doug

A comparison of two approaches to evaluating search quality: query-based evaluation (aggregating clicks per query into relevance labels) vs. session-based evaluation (replaying individual user sessions). Session-based eval offers better sampling accuracy by treating each user interaction equally, similar to probability-based polling, and preserves time-sensitive features like dynamic pricing for learning-to-rank training. However, it sacrifices per-query debuggability. The post recommends using both approaches for different purposes: session-based for simulated A/B testing and query-based for diagnosing specific query failures.

Session vs Query based search evals

But there’s a different way - session based eval