The engineering behind Tinybird branches: zero-copy partition sharing via ClickHouse metadata, Kafka consumer group isolation, S3 sample imports, and CI/CD preview deployments.

Tinybird is a real-time data analytics platform for building fast and scalable data pipelines, offering tools for data ingestion, processing, and visualization. From this blog users can learn about stream processing, data warehousing, and analytics techniques to unlock insights from their data and make data-driven decisions.

Tinybird

Tinybird's engineering team explains how they built branch environments for their data platform. Branches are ephemeral ClickHouse databases that share production data without copying it, using ClickHouse's hardlink mechanism via ALTER TABLE ATTACH PARTITION FROM — pointing branch tables at the same S3 objects as production with reference counting, not duplication. Kafka connectors in branches get isolated via new consumer groups starting at the latest offset, while S3/GCS connectors use a sample API that imports recent files without touching the production sync cursor. Schema migrations to production are handled by tb deploy, which builds a dependency graph, runs backfills with UNION views for zero-downtime transitions, then swaps metadata. The entire workflow is CLI/API-first, making it accessible to both developers and coding agents.

How We Built Branches: Sharing Production Data Without Copying It