“To make our logging system work, we had to make a lot of choices. The key is how you simplify things in order to do the least amount of work.”  Daniel Muino, Software Engineer

ClickHouse is a leading platform for open-source columnar database technology and real-time analytics. With ClickHouse, developers can delve into documentation, case studies, and performance benchmarks to learn how to build highly scalable and performant analytical systems. It provides tips about ClickHouse's storage engine, query execution, and distributed architecture. Developers can gain insights into their data in real-time, enabling them to make informed decisions and extract  business insights.

ClickHouse

Netflix processes 5 petabytes of logs daily using ClickHouse, handling 10.6 million events per second with sub-second query performance. Three key optimizations enabled this scale: replacing regex-based log fingerprinting with generated lexers (8-10x faster), implementing custom native protocol serialization for efficient data ingestion, and sharding tag maps to reduce query times from 3 seconds to 700ms. The system combines ClickHouse for hot data with Apache Iceberg for long-term storage, making logs searchable within 20 seconds while serving 500-1,000 queries per second across 40,000+ microservices.

How Netflix optimized its petabyte-scale logging system with ClickHouse

Inside Netflix’s logging architecture #

Optimization #1: Ingestion - Fingerprinting #

<p>im just thinking how big company like netflix do their end to give best user expirence as posible to their users</p>
