Instacart developed LACE (LLM-Assisted Chatbot Evaluation), an automated framework to evaluate customer support chatbot performance at scale. The system uses three evaluation methods: direct prompting, agentic reflection, and agentic debate with multiple AI agents. LACE evaluates conversations across five dimensions: query understanding, answer correctness, chat efficiency, client satisfaction, and compliance. Through iterative refinement and human-AI alignment validation, the framework identifies chatbot issues and drives continuous improvements in customer support quality.
Table of contents
Turbocharging Customer Support Chatbot Development with LLM-Based Automated EvaluationIntroductionDefining Evaluation CriteriaBuilding a Robust Evaluation EngineAligning Model and Human JudgmentWhat We Learned from Iterative RefinementLACE in Action: Driving Continuous ImprovementSort: