Static benchmarks made sense for static software. Agents that adapt to users, rewrite their own harnesses, and shift behavior over time break that assumption. This talk is about what evaluation looks like when the system you're measuring keeps changing underneath you.

Vincent Koc traces the arc from prompt engineering to context engineering to intent engineering, where agents self-optimize toward what users actually want. The eval problem compounds at each step: production traces reveal behavioral drift, test suites go stale, and the 20% of edge cases that break your product rarely show up in handcrafted datasets. The alternative he proposes: define the end state, let agents curate their own suites from traces, and treat evals as a living system rather than a point-in-time snapshot.

Speaker info:
- https://x.com/vincent_koc

AI Engineer

A conference talk by Vincent Koc from Comet ML arguing that traditional static AI benchmarks and evaluations are insufficient for modern agentic AI systems. He traces the evolution from prompt engineering to context engineering to 'intent engineering,' where AI systems self-optimize based on user intent. The core argument is that evals need to become adaptive and living systems rather than static datasets — using agent traces, always-on evaluation, and telemetry-in-the-loop to continuously update test suites as applications and user behavior evolve. He introduces the concept of 'eval calcification' to describe the growing gap between static benchmarks and dynamic agentic behavior, and advocates for treating evaluations as self-optimizing agents rather than fixed data points.

Dark Factory: OpenClaw Ships Faster Than You Can Read the Diff — Vincent Koc, Comet ML