Best of Data Streaming — November 2024

1. 1
   Article

   Trendyol Tech·1y

   Ensuring Client Continuity in Kafka: Handling Broker Restarts with No Disruptions

   Trendyol's Data Streaming team addresses challenges in maintaining uninterrupted Kafka services by leveraging Confluent Stretch Kafka across multiple data centers. The team ensures high availability and fault tolerance by configuring replication factors and monitoring topic configurations. By implementing custom alert mechanisms and offering different topic creation options, they reduce downtime and errors during broker restarts, ensuring client applications remain unaffected.

   15
2. 2
   Article

   Collections·2y

   How LinkedIn Scaled Their System to 5 Million Queries Per Second

   LinkedIn scaled their Restrictions and Enforcement System to handle 5 million queries per second by using advanced techniques such as BitSets, Bloom Filters, and full refresh-ahead caching strategies. The architecture includes components like the Venice Database and Kafka for real-time data streaming, ensuring high availability, low latency, and efficient memory usage.

   11
3. 3
   Article

   Data Engineer Things·2y

   I spent 8 hours researching WarpStream

   WarpStream, a novel messaging system introduced in 2023, aims to address the challenges of managing Kafka infrastructure, particularly in cloud environments. It operates with a Bring Your Own Cloud (BYOC) model, using stateless agents and object storage to reduce operational overhead and costs. WarpStream's architecture separates the data and control planes, ensuring data privacy and efficient, scalable performance. Despite some latency trade-offs and current limitations in feature support compared to Kafka, WarpStream provides a cost-effective streaming solution for many use cases.

   10