How we run ClickHouse on top of object storage with local SSD caching, zero-copy replication, and a packed part format that cuts S3 costs by 30-40%.

Tinybird is a real-time data analytics platform for building fast and scalable data pipelines, offering tools for data ingestion, processing, and visualization. From this blog users can learn about stream processing, data warehousing, and analytics techniques to unlock insights from their data and make data-driven decisions.

Tinybird

Tinybird's storage architecture separates compute from storage using S3/GCS as the primary data store with local SSD caching on each replica. Key components include the Gatherer (batches streaming events before writing to avoid small-file proliferation), zero-copy replication (replicas share one S3 copy rather than duplicating data), and a custom packed part format developed in Tinybird's ClickHouse fork that consolidates MergeTree metadata files into a single S3 object, reducing write API calls by 30-40%. The upstream ClickHouse project deprecated zero-copy replication, which is a primary reason Tinybird maintains its own fork. Cache eviction is managed by ClickHouse with safeguards to prevent large scans from evicting hot data.

How Tinybird's storage architecture works: S3, local caching, and zero-copy replication