Pushing the Frontier for Data Agents with Genie

Databricks' Genie data agent addresses unique challenges of enterprise data environments that coding agents struggle with, including dynamic data discovery across massive lakehouses, ambiguous ground truth, and complex cross-system reasoning. Three key innovations drive its performance: (1) Specialized Knowledge Search using semantic indices and metadata signals to improve table discovery by up to 40%, (2) Parallel Thinking that samples multiple reasoning trajectories and aggregates results to improve accuracy without reliable unit tests, and (3) Multi-LLM design that assigns different frontier or open-source models to different sub-agents (planning, search, code generation, judging) based on their complementary strengths. Combined, these techniques improve accuracy from 32% to over 90% on internal benchmarks compared to a leading coding agent baseline, while also reducing cost and latency through methods like GEPA prompt optimization.