Tinybird built a custom Kafka to ClickHouse connector to overcome limitations of ClickHouse's built-in Kafka engine, including partial Schema Registry support and scaling challenges. The Python-based service uses a controller/worker pattern for horizontal scaling and fault isolation, with rendezvous hashing for topic assignment
•10m read time• From tinybird.co
Table of contents
Why build our own Kafka connector?Architecture of a scalable Kafka connectorFive design decisions that make it production readyHow we scale and monitor billions of eventsGetting StartedWhat we learnedSort: