unknown

more evals != better agents

at @LangChain, every eval our team maintains is like a noisy vector that shapes agent behavior in production

as teams hill climb hundreds or thousands of evals to “improve their agents”, this problem of bad evals contributing noisy behavior compounds

here’s a practical guide on:
- how we design small subsets of targeted evals that test important abilities for our Task
- our team’s shared responsibility approach for justifying every eval’s value and what it measures
- metrics that influence our decisions beyond accuracy
- how we store and run evals

<p>more evals != better agents

at @LangChain, every eval our team maintains is like a noisy vector that shapes agent behavior in production

as teams hill climb hundreds or thousands of evals to “improve their agents”, this problem of bad evals contributing noisy behavior compounds

here’s a practical guide on:
- how we design small subsets of targeted evals that test important abilities for our Task
- our team’s shared responsibility approach for justifying every eval’s value and what it measures
- metrics that influence our decisions beyond accuracy
- how we store and run evals</p>

Harrison Chase

A tweet arguing that adding more evaluations does not necessarily lead to better AI agents, suggesting that each eval maintained by a team acts as a noisy vector influencing agent behavior, implying quality and signal clarity matter more than quantity.