Best of Kafka — February 2025

1. 1
   Article

   Towards AI·1y

   End-to-End Data Engineering System on Real Data with Kafka, Spark, Airflow, Postgres, and Docker

   The post provides a detailed guide on building an end-to-end data engineering system using Kafka for data streaming, Spark for data transformation, Airflow for orchestration, PostgreSQL for storage, and Docker for setup and deployment. It is structured into two phases: the first focuses on constructing the data pipeline, while the second will cover creating an application to interact with the database using language models. This project is particularly suited for beginners to data engineering, aiming to deepen their practical knowledge of handling data systems.

   99
2. 2
   Article

   Last9·1y

   Top 13 Kafka Monitoring Tools You Should Know

   Running Kafka in production requires effective performance monitoring to detect bottlenecks and troubleshoot issues. Key metrics to monitor include broker health, cluster performance, consumer health, producer performance, and ZooKeeper. The post reviews 13 Kafka monitoring tools, including Prometheus & Grafana, Last9, Confluent Control Center, and Datadog, detailing their features, strengths, and user feedback. It concludes by helping you choose the right tool based on your needs.

   15
3. 3
   Article

   Trendyol Tech·1y

   Kafka Consumer Monitoring Tips

   Microservice architecture relies heavily on message brokers like Kafka, making consumer monitoring essential for business sustainability. This post discusses how Trendyol monitors Kafka consumers using Prometheus and handles alerts through Alert Manager. It covers key issues like lag, disconnected consumers, chronic rebalancing, skewed partitions, idle consumers, and consumer processing metrics, providing detailed approaches and PromQL expressions to detect and address these problems.

   11