Genie is Uber’s internal on-call copilot, designed to provide real-time support for thousands of queries across multiple help channels in Slack®. It enables users to receive prompt responses with proper citations from Uber’s internal documentation. It also improves the productivity of on-call engineers and subject matter experts (SMEs) by reducing the effort required to address common, ad-hoc queries. While Genie streamlines the development of an LLM-powered on-call Slack bot, ensuring the accuracy and relevance of its responses remains a significant challenge. This blog details our efforts to improve Genie’s answer quality to near-human precision, allowing SMEs to rely on it for most queries without concern over potential misinformation in the engineering security and privacy domain.

The Uber Engineering Blog offers insights, technical deep dives, and updates on the engineering challenges and solutions behind Uber's platform and services. Covering topics such as distributed systems, data infrastructure, and mobile development, the blog provides resources for developers interested in large-scale systems and real-world engineering problems. Developers can learn about Uber's technology stack, engineering culture, and best practices for building scalable and reliable systems.

Uber Engineering

Uber developed Enhanced Agentic-RAG (EAg-RAG) to improve their internal chatbot Genie's accuracy from traditional RAG systems. The approach includes enriched document processing with custom Google Docs loaders, LLM-powered agents for query optimization and source identification, and automated evaluation using LLM-as-Judge. This resulted in a 27% relative increase in acceptable answers and 60% reduction in incorrect advice for engineering security and privacy queries.

Enhanced Agentic-RAG: What If Chatbots Could Deliver Near-Human Precision?