How We Built Branches: Sharing Production Data Without Copying It

Tinybird's engineering team explains how they built branch environments for their data platform. Branches are ephemeral ClickHouse databases that share production data without copying it, using ClickHouse's hardlink mechanism via ALTER TABLE ATTACH PARTITION FROM — pointing branch tables at the same S3 objects as production with reference counting, not duplication. Kafka connectors in branches get isolated via new consumer groups starting at the latest offset, while S3/GCS connectors use a sample API that imports recent files without touching the production sync cursor. Schema migrations to production are handled by tb deploy, which builds a dependency graph, runs backfills with UNION views for zero-downtime transitions, then swaps metadata. The entire workflow is CLI/API-first, making it accessible to both developers and coding agents.