Evaluating Netflix Show Synopses with LLM-as-a-Judge by Gabriela Alessio, Cameron Taylor, and Cameron R. Wolfe Introduction When members log into Netflix, one of the hardest choices is what to watch …

The Netflix Tech Blog provides insights into the engineering practices, architecture, and innovations behind the Netflix streaming platform. Developers can learn about Netflix's microservices architecture, data infrastructure, and machine learning algorithms, as well as explore topics such as video encoding, content delivery, and personalization algorithms used in the Netflix service.

Netflix TechBlog

Netflix's engineering team describes their LLM-as-a-Judge system for automatically evaluating show synopsis quality at scale. The system scores synopses across four quality dimensions (tone, clarity, precision, factuality) using a combination of techniques: per-criteria dedicated judges, tiered rationales (extended reasoning with concise summaries), consensus scoring (5× sampling with aggregation), and an Agents-as-a-Judge approach for factuality detection. The system achieves 85%+ agreement with expert creative writers. Validation against member behavior shows that higher LLM quality scores correlate with better take fraction and lower abandonment rates, confirming the system captures signals meaningful to real users. Key lessons include: single prompts evaluating multiple criteria hurt accuracy, longer rationales improve scoring but hurt readability (solved by tiered rationales), and reasoning models offer marginal gains at high cost.

Evaluating Netflix Show Synopses with LLM-as-a-Judge

Scaling Quality Scoring with LLM-as-a-Judge

Get Netflix Technology Blog ’s stories in your inbox