Migrating from Postgres to ScyllaDB, with 349X Faster Query Processing

This title could be clearer and more informative.Try out Clickbait Shieldfor free (5 uses left this month).

Coralogix migrated their metadata store from PostgreSQL to ScyllaDB, achieving a 349x performance improvement by reducing query processing time from 30 seconds to 86 milliseconds. The migration involved redesigning data models for NoSQL, implementing bloom filter chunking strategies, and optimizing partition keys. Despite challenges with data modeling and EBS storage decisions, the team successfully deployed a 3-node ScyllaDB cluster handling 10K writes per second with sub-millisecond P99 latency, processing terabytes of Parquet metadata for their observability platform.

11m read timeFrom p99conf.io
Post cover image
Table of contents
Metastore Motivation and RequirementsThe Initial PostgreSQL ImplementationScyllaDB ImplementationLessons LearnedFuture Plans: WebAssembly UDFs with RustWatch the Complete Tech Talk
6 Comments

Sort: