Tinybird built a custom Kafka to ClickHouse connector to overcome limitations of ClickHouse's built-in Kafka engine, including partial Schema Registry support and scaling challenges. The Python-based service uses a controller/worker pattern for horizontal scaling and fault isolation, with rendezvous hashing for topic assignment
Table of contents
Why build our own Kafka connector?Architecture of a scalable Kafka connectorFive design decisions that make it production readyHow we scale and monitor billions of eventsGetting StartedWhat we learnedSort: