Learn how Arctic Wolf used liquid clustering to deliver faster, fresher security analytics at petabyte scale, enabling extended and responsive threat insights for our customers.

databricks

Arctic Wolf processes over 1 trillion security events daily (60+ TB compressed) and needed faster query performance on 3.8+ PB of data. By migrating from date-hour partitioning with z-ordering to liquid clustering with Unity Catalog managed tables, they reduced file counts from 4M to 2M, cut query times by ~50% (90-day queries from 51s to 6.6s), and improved data freshness from hours to minutes. The architecture uses medallion design with Kafka ingestion, structured streaming with clustering-on-write, and batching via maxBytesPerTrigger to maintain optimal file layouts without constant OPTIMIZE operations, enabling near-real-time threat detection at petabyte scale.

Arctic Wolf’s Liquid Clustering Architecture Tuned for Petabyte Scale

Legacy Bottlenecks: Why Arctic Wolf Rebuilt

Building the Streaming Data Foundation with liquid clustering