Hello, I’m @otter, a software engineer working in the search domain at Mercari.This article is the entry for Day 9

Mercari's platform is a resource for developers and technology enthusiasts, offering insights into engineering challenges, technology stack, and culture at Mercari. Through articles, tech talks, and open-source contributions, Mercari offers insights into building innovative and scalable technology solutions. Developers can learn about Mercari's engineering culture, agile methodologies, and contributions to the broader tech community.

Mercari Engineering

Mercari automated their search results quality monitoring by replacing manual review processes with LLM-based evaluation. Using Gemini 2.5 Pro, they implemented a system that scores search result relevance on a 0.0-1.0 scale based on Amazon's ESCI criteria (Exact, Substitute, Complement, Irrelevant). The solution provides both online monitoring of production queries (1,000 keywords weekly, 120 items each) and offline evaluation tools for pre-release testing. By combining product metadata with image data and standardizing evaluation prompts, they achieved consistent, objective quality assessments that replaced subjective manual reviews.

Search Results Quality Monitoring with LLMs

Mercari’s Product Search and Its Quality Management

Challenges and Requirements in Search Results Quality Review

Achieving Objective and Stable Monitoring with LLMs and Evaluation Criteria