Pinterest launched a next-generation CDC-based database ingestion framework using Kafka, Flink, Spark, and Iceberg. The system reduces data availability latency from 24+ hours to 15 minutes, processes

InfoQ is a leading online platform for software developers, architects, and technical leaders, providing news, articles, presentations, and interviews on a wide range of topics, including agile practices, DevOps, microservices, and emerging technologies. With a focus on quality content and expert insights, InfoQ helps professionals stay informed about the latest trends, best practices, and industry developments. Developers can learn from real-world experiences, gain  knowledge, and connect with peers in the global software community through InfoQ's diverse and engaging content.

InfoQ

Pinterest replaced its legacy batch-based database ingestion system with a CDC-powered framework using Debezium/TiCDC, Kafka, Flink, Spark, and Apache Iceberg. The new architecture reduces data availability latency from over 24 hours to 15 minutes by processing only changed records (roughly 5% of daily data) rather than full-table scans. CDC tables act as append-only ledgers with sub-5-minute latency, while base tables are updated via Spark Merge Into operations every 15–60 minutes. Pinterest standardized on Iceberg's Merge on Read strategy over Copy on Write to control storage costs at petabyte scale. Additional optimizations include Iceberg bucketing by primary key hash to parallelize upserts and reduce small-file overhead. The system supports MySQL, TiDB, and KVStore, is configuration-driven, and spans thousands of pipelines. Future work targets automated schema evolution.

Pinterest’s CDC-Powered Ingestion Slashes Database Latency from 24 Hours to 15 Minutes