databricks

Databricks introduces Real-Time Mode (RTM) for Apache Spark Structured Streaming, enabling sub-second latencies without requiring a second streaming engine like Apache Flink. RTM eliminates the need for dual data processing stacks by evolving Spark's execution engine with three key technical innovations. It allows teams to use the same Spark API for both high-throughput ETL and ultra-low-latency applications, reducing logic drift between training and inference, enabling latency changes with a single-line code change, and simplifying operational overhead. Benchmarks show performance comparable to Flink. Early adopters include DraftKings for fraud detection, e-commerce platforms for personalization, and IoT monitoring use cases.

Real-Time Mode: Ultra-low latency streaming on Spark APIs without a second engine

Teams operate lesser infrastructure, and move faster with Spark

Customers are running several real-time applications on Spark