High-performance full-text search, even on object storage. This deep dive from the ClickHouse engineers explains the new text index design and how it keeps queries fast at scale.

ClickHouse is a leading platform for open-source columnar database technology and real-time analytics. With ClickHouse, developers can delve into documentation, case studies, and performance benchmarks to learn how to build highly scalable and performant analytical systems. It provides tips about ClickHouse's storage engine, query execution, and distributed architecture. Developers can gain insights into their data in real-time, enabling them to make informed decisions and extract  business insights.

ClickHouse

ClickHouse engineers detail the redesigned full-text search index built to perform efficiently on object storage. The new design uses a block-based dictionary with front-coding compression, a sparse in-memory index for fast block lookup, and three posting list representations (Roaring Bitmaps, VarInt-encoded, and embedded) chosen adaptively by cardinality. Sequential access patterns replace the old FST-based random-read approach, enabling efficient merges and queries on remote storage. Query execution supports three modes: direct read (posting lists resolve queries without touching the text column), direct read with hint (posting lists narrow candidates before LIKE evaluation), and fallback granule-skipping. Configuration covers tokenizers (splitByNonAlpha, splitByString, ngrams, sparseGrams, array) and a preprocessor pipeline for lowercasing, accent removal, and HTML stripping. Benchmarks show over 7x speedup on array-tag queries. Upcoming work includes JSON column indexing, phrase search, and faster regex evaluation.

Building high-performance full-text search for object storage

How the query engine uses the text index #

What this means for ClickHouse Cloud users #