Best of Kafka — May 2024

1. 1
   Article

   ByteByteGo·2y

   Cloudflare’s Trillion-Message Kafka Infrastructure: A Deep Dive

   Cloudflare's Kafka infrastructure has processed 1 trillion messages, demonstrating significant scaling and resilience. The engineering team faced challenges with coupling, unstructured communication, and common usage patterns. Lessons learned include the importance of balance between configuration and simplicity, visibility in distributed systems, well-defined contracts, and knowledge sharing.

   62
2. 2
   Article

   DEV·2y

   Integrate Kafka with a Serverless application

   Learn how to integrate Apache Kafka with a Serverless application in the AWS ecosystem using event-driven architecture, Amazon EventBridge, and managed services.

   33
3. 3
   Article

   Javarevisited·2y

   No More ZooKeeper

   Kafka is removing its dependency on ZooKeeper, which has been an integral part of Kafka clusters for metadata management. The motivation behind this change is to overcome limitations such as single point of failure, scalability issues, high recovery time, increased resource utilization, and complexity. Instead, Kafka will use a self-managed metadata quorum called KRaft. In KRaft mode, leader selection works by electing a leader through a voting process.

   10

   1