From Print to Digital: Making Weekly Flyers Shoppable at Instacart Through Computer Vision and LLMs

Instacart built a two-phase ML pipeline to automate digitization of grocery flyers, reducing processing time from 3-4 hours to 30 minutes. Phase 1 uses Meta's Segment Anything Model with custom algorithms for image segmentation, achieving 75-90% accuracy in extracting product bounding boxes. Phase 2 combines OCR, LLMs, and Instacart's existing search infrastructure to match segmented images to catalog products with 95% recall. The system handles complex flyer layouts through techniques like Weighted Boxes Fusion, model ensembling, and ML-based filtering.