A great example of why timely dataflow's approach to progress tracking can be orders of magnitude more efficient than other stream processors.

Materialize Blog offers insights, tutorials, and updates on real-time data processing, streaming analytics, and event-driven architectures. Covering topics such as Apache Kafka, distributed systems, and stream processing frameworks, Materialize Blog provides resources for data engineers and streaming enthusiasts. Developers can learn about building scalable data pipelines, processing real-time data streams, and leveraging streaming analytics for actionable insights through Materialize's blog posts and tutorials.

Materialize

Frank McSherry demonstrates a 100x speedup in Timely Dataflow by introducing a third operator scheduling mode: 'notify only if holding a capability'. In a benchmark with 1,000 dataflows each containing ~1,000 operators, the naive approach required visiting all ~1,000,000 operators on every tick (similar to Flink's operator-to-operator progress propagation), taking ~350ms per iteration. By allowing REGION and ARRANGE operators to opt out of timestamp notifications when they hold no pending work, the system drops to ~4ms per iteration. The key insight is that Timely Dataflow tracks progress at the system level via timestamp capabilities rather than operator-to-operator communication, enabling it to skip dormant operators entirely. This matters especially for real-world deployments like Materialize's catalog server, which has ~12,000 operators across ~100 dataflows where most are idle at any given moment.

Speeding up Timely Dataflow by 100x

Smartness: tracking progress in timely dataflow