How and Why Netflix Built a Real-Time Distributed Graph: Part 1 — Ingesting and Processing Data Streams at Internet Scale Authors: Adrian Taruc and James Dalton This is the first entry of a …

The Netflix Tech Blog provides insights into the engineering practices, architecture, and innovations behind the Netflix streaming platform. Developers can learn about Netflix's microservices architecture, data infrastructure, and machine learning algorithms, as well as explore topics such as video encoding, content delivery, and personalization algorithms used in the Netflix service.

Netflix TechBlog

Netflix built a Real-Time Distributed Graph (RDG) to analyze member interactions across different business verticals like streaming, gaming, and live events. The system processes over 1 million Kafka messages per second using Apache Flink jobs that transform events into graph nodes and edges, writing more than 5 million records per second to storage. The architecture evolved from a monolithic Flink job to a 1:1 mapping between Kafka topics and Flink jobs for better operational stability and tuning. This first part covers the ingestion and processing pipeline, with future posts planned for storage and serving layers.

How and Why Netflix Built a Real-Time Distributed Graph: Part 1 — Ingesting and Processing Data Streams at Internet Scale

Get Netflix Technology Blog’s stories in your inbox

From One Job to Many: Scaling Flink the Hard Way