Best of Kafka — June 2025

1. 1
   Article

   Medium·50w

   How Kafka Saved Our Payment System And Helped Us Scale to 10 Million Users

   A payment system was failing due to synchronous processing of multiple tasks (email, notifications, logging) in a single thread, causing delays and duplicate charges. The team implemented Kafka as a message broker to decouple services through event-driven architecture. After a payment succeeds, the system publishes a single event to Kafka, allowing independent services to consume and process it asynchronously. This approach eliminated blocking operations, improved response times, reduced support tickets, and enabled the system to scale to 10 million users while maintaining reliability and making it easier to add new features.

   144

   5
2. 2
   Article

   Medium·47w

   Why We Replaced Kafka with gRPC for Service Communication

   A development team replaced Kafka with gRPC for synchronous service communication in their loan servicing platform after experiencing issues with debugging, latency, and operational complexity. While keeping Kafka for appropriate use cases like audit logs and fan-out patterns, they found gRPC provided better performance (70-80% latency reduction), easier debugging, and simpler infrastructure management for request-response interactions. The key lesson was using each tool for its intended purpose rather than forcing one solution everywhere.

   135

   7
3. 3
   Article

   Data Engineer Things·47w

   Building a Real-Time Flight Data Pipeline with Kafka, Spark, and Airflow

   A comprehensive guide to building a real-time flight data pipeline using Kafka for streaming, Spark for processing, and Airflow for orchestration. The pipeline fetches live flight data from a custom API, streams it through Kafka to MongoDB for storage, then uses Airflow to schedule daily ETL jobs that extract landed flight information into PostgreSQL and generate CSV reports. The project includes Docker containerization, complete code examples, and demonstrates end-to-end data engineering practices from real-time ingestion to batch processing and reporting.

   119

   4
4. 4
   Video

   Community Picks·48w

   Learn Microservices and Kafka with an E-commerce Example | Kafka Tutorial for beginners

   A comprehensive tutorial demonstrating how to transform a monolithic e-commerce application into a microservices architecture using Apache Kafka for inter-service communication. The guide covers breaking down payment, order, email, and analytics services into independent components, implementing Kafka producers and consumers, setting up Docker containers, and creating fault-tolerant Kafka clusters with multiple brokers and partitions. The tutorial includes practical code examples showing how to handle asynchronous messaging, reduce response times from 12 seconds to 3 seconds, and ensure system resilience through distributed architecture.

   101
5. 5
   Article

   Data Engineer Things·49w

   Stream Kafka Topic to the Iceberg Tables with Zero-ETL

   AutoMQ introduces Table Topic, an open-source feature that automatically converts Kafka topic messages to Iceberg tables without requiring separate ETL pipelines. The solution addresses the complexity of managing Kafka-to-lakehouse data flows by handling schema management, partitioning, and upsert operations automatically. This represents an evolution from Kafka's original shared-nothing architecture to a shared-data approach, where data is accessible through both Kafka APIs and as Iceberg tables for analytics workloads.

   10