LinkedIn Engineering

LakeChime is a data trigger service that unifies data trigger semantics for modern and traditional table formats in data lakes. It supports both partition triggers and snapshot triggers, providing backward compatibility with Hive and forward compatibility with modern table formats. LakeChime enables simpler data lake migrations, incremental compute, and ease of integration with data producers, consumers, and scheduling systems. The key foundation of LakeChime's data trigger system is the Data Change Event (DCE), which captures data changes within a data table. Users can leverage LakeChime's partition and snapshot triggers to tailor their data processing workflows. Furthermore, LakeChime integrates with Airflow to facilitate efficient incremental processing with Spark on Iceberg tables.

LakeChime: A Data Trigger Service for Modern Data Lakes

The Evolution and Impact of Table Formats on Data Trigger Mechanisms

Introducing LakeChime: A Unified Data Trigger Solution

Data Change Event: The Foundation of LakeChime's Data Trigger System