Append-only tables and incremental reads — Jack Vanlightly

The post discusses the support for append-only tables and incremental reads in various table formats such as Apache Iceberg, Delta Lake, Apache Hudi, and Apache Paimon. It explains how incremental reads allow compute engines to return new records or changes since the last query. Each table format supports these features differently, with Iceberg and Delta adding new data files without performing data conflict checks, whereas Hudi uses file groups and Paimon uses row-level operations. The post also touches on the performance implications and potential data conflicts with multiple writers.