Human judgment in the agent improvement loop

Building reliable AI agents requires capturing tacit organizational knowledge through a continuous improvement loop. Using a financial trader copilot as a running example, the post outlines how human judgment should inform workflow design, tool design, and agent context. The recommended approach centers on translating expert knowledge into automated LLM-as-a-judge evaluators rather than relying on manual review at scale. The improvement loop covers three phases: initial development with curated test suites, post-deployment monitoring with online evaluations and annotation queues, and continuous refinement by turning production traces into better test datasets. LangSmith features like Align Evaluator, tracing, automations, and Insights Agent are used throughout to operationalize this process.