Pinterest Search scaled their relevance assessment by fine-tuning open-source multilingual LLMs (XLM-RoBERTa-large) on human-annotated data to predict search result relevance. This approach reduced labeling costs and time while achieving 73.7% exact match with human labels and strong correlation metrics (Kendall's τ>0.5). By
•9m read time• From medium.com
Table of contents
IntroductionMethodologyFine-tuned LLMs as Relevance ModelStratified Sampling DesignRelevance Measurement with LLMsResultsSummaryFuture WorkAcknowledgementSort: