Apache Iceberg was built for batch analytics — but operational data changes continuously. Learn how Materialize streams live, transactionally consistent data into Iceberg without the memory and latency costs of batching.

Materialize Blog offers insights, tutorials, and updates on real-time data processing, streaming analytics, and event-driven architectures. Covering topics such as Apache Kafka, distributed systems, and stream processing frameworks, Materialize Blog provides resources for data engineers and streaming enthusiasts. Developers can learn about building scalable data pipelines, processing real-time data streams, and leveraging streaming analytics for actionable insights through Materialize's blog posts and tutorials.

Materialize

Materialize's Field CTO explains the engineering challenges of streaming live, transactionally consistent operational data into Apache Iceberg, which was designed for batch ETL. Key problems addressed include: avoiding memory-heavy buffering by minting batch descriptions ahead of time so workers can stream data to S3 immediately; handling CDC deletes efficiently using position deletes (cheap) vs. equality deletes (expensive) with an in-memory primary key map; storing recovery state directly in Iceberg snapshot properties to avoid external coordination systems; managing thousands of empty snapshots needed to track frontier progress; and the unsolved problem of multi-table transactional consistency, which Iceberg's spec doesn't currently support.

Making Iceberg Work for Operational Data