How we built a scalable, reliable Kafka connector that processes billions of events with minimal operational overhead

Tinybird is a real-time data analytics platform for building fast and scalable data pipelines, offering tools for data ingestion, processing, and visualization. From this blog users can learn about stream processing, data warehousing, and analytics techniques to unlock insights from their data and make data-driven decisions.

Tinybird

Tinybird built a custom Kafka to ClickHouse connector to overcome limitations of ClickHouse's built-in Kafka engine, including partial Schema Registry support and scaling challenges. The Python-based service uses a controller/worker pattern for horizontal scaling and fault isolation, with rendezvous hashing for topic assignment and circuit breaker patterns for reliability. Key optimizations include RowBinary encoding via a custom C++ library (30% faster ingestion), an ingestion proxy to prevent TOO_MANY_PARTS errors, and native libraries (librdkafka, orjson, fastavro) for performance-critical operations. The connector processes billions of events daily with KEDA autoscaling, comprehensive observability through ClickHouse ops logs and Prometheus metrics, and enterprise features like Schema Registry integration, automatic metadata injection, and quarantine tables for failed messages.

How we built a production Kafka connector for ClickHouse®

Architecture of a scalable Kafka connector

Five design decisions that make it production ready

How we scale and monitor billions of events