Our Early Journey to Transform Instacart’s Discovery Recommendations with LLMs

Instacart's ML team shares how they rebuilt their Shopping Hub recommendation system using LLMs. The new AI-native platform uses a top-down, cascaded generation approach: a page design agent generates personalized themes from user context, a fine-tuned student model generates retrieval keywords via RAG, and quality/diversity filtering guards against off-brand or redundant content before passing to existing ranking infrastructure. Key techniques include teacher-student fine-tuning on Llama/Qwen models, RAG-based keyword candidate pruning (reducing generation costs 15-20%), and a fine-tuned DeBERTa cross-encoder for scalable quality filtering (99% cost reduction vs. LLM inference). Early A/B results are promising, with generative placements outperforming static baselines in offline evaluations. Key learnings: keep modeling tasks focused, invest heavily in evals, and add structure to input/output layers.