We build AI agents at Databricks, both internal agents that power our own engineering workflows and customer-facing agents that ship as part of the Databricks platform. These agents can perform long-running tasks, generate thousands of lines of code, and create new data and AI assets among other things. While we had some basic checks in place early on, we lacked the kind of comprehensive, automated test suite that would let us iterate with confidence. This post describes how we bridged that gap using MLflow and a set of best practices around it, which we refer to as coSTAR (coupled Scenario, Trace, Assess, Refine): : two loops working in concert, one that refines the judges until they reflect human expert judgment, and one that refines the agent until it passes those judges.

databricks

Databricks engineers describe coSTAR, a methodology for testing and iteratively improving AI agents using MLflow. The framework draws a direct analogy to traditional software testing: scenario definitions act as test fixtures, MLflow traces serve as the test harness, LLM-based judges replace assertions, and a coding assistant drives the refine loop. A key innovation is the dual-loop design — one loop aligns judges against human expert golden sets, and a second loop uses those trusted judges to automatically refine the agent. The post covers agentic judges (judges that use tools to inspect traces selectively rather than consuming full traces), deterministic checks, operational metrics, regression testing for MCP tool dependencies, and reusing the same judges for production monitoring. Known limitations include manual scenario generation, judge alignment cost, overfitting risk, and multi-step failure attribution.

coSTAR: How We Ship AI Agents at Databricks Fast, Without Breaking Things

Regression Tests for Infrastructure, Not Just the Agent