Wix Engineering

Wix built a Data-to-Production platform that bridges their petabyte-scale Iceberg data warehouse with production microservices requiring millisecond latency. The system consists of three layers: a self-service metadata registration console with human-in-the-loop governance, a dynamic Airflow-based ingestion engine supporting three loading strategies (OVERWRITE, UPSERT, PARTITION_REPLACE) with custom idempotent operators, and a type-safe Scala serving layer using a semantic JSON DSL that compiles to optimized ClickHouse SQL. The platform handles billions of rows, automatic schema evolution, catch-up logic, and includes safety mechanisms like empty staging protection and row count validation, while preventing SQL injection and enforcing resource governance.

Data-to-Production: Bridging the Gap Between Iceberg and Live Microservices

1. The Metadata Layer: Registration & Governance

2. The Engine Room: Deep Dive into the Ingestion Pipeline

3. The Serving Layer: Safe, Semantic, and Low-Latency Access

3.3. Performance Optimizations & Consistency