Instacart developed LACE (LLM-Assisted Chatbot Evaluation), an automated framework to evaluate customer support chatbot performance at scale. The system uses three evaluation methods: direct prompting, agentic reflection, and agentic debate with multiple AI agents. LACE evaluates conversations across five dimensions: query
Table of contents
Turbocharging Customer Support Chatbot Development with LLM-Based Automated EvaluationIntroductionDefining Evaluation CriteriaBuilding a Robust Evaluation EngineAligning Model and Human JudgmentWhat We Learned from Iterative RefinementLACE in Action: Driving Continuous ImprovementSort: