Best of ETL — June 2025

1. 1
   Article

   databricks·50w

   Announcing Lakeflow Designer: No-Code ETL, Powered by the Databricks Intelligence Platform

   Databricks introduces Lakeflow Designer, an AI-powered no-code pipeline builder that enables business analysts to create production-ready ETL pipelines without coding. The tool generates standard Lakeflow Declarative Pipelines that data engineers can review and modify, eliminating the typical separation between business and technical teams. Designer leverages AI grounded in organizational data context and provides built-in governance, observability, and scalability within the unified Databricks platform.

   13

   1
2. 2
   Article

   Programming Digest·47w

   Which Data Architecture Should I Choose for My Workplace? — A Data Engineer’s Approach

   A comprehensive guide comparing four major data architecture approaches: Data Warehouse, Data Lake, Data Lakehouse, and Data Mesh. The article explains when to use each approach, their advantages and challenges, and provides platform recommendations. It focuses on the Medallion Architecture with its Bronze, Silver, and Gold layers for modern data warehouse design, emphasizing the importance of requirement analysis and proper architectural selection based on data types, analytical needs, and organizational structure.

   12
3. 3
   Article

   Data Engineer Things·49w

   Stream Kafka Topic to the Iceberg Tables with Zero-ETL

   AutoMQ introduces Table Topic, an open-source feature that automatically converts Kafka topic messages to Iceberg tables without requiring separate ETL pipelines. The solution addresses the complexity of managing Kafka-to-lakehouse data flows by handling schema management, partitioning, and upsert operations automatically. This represents an evolution from Kafka's original shared-nothing architecture to a shared-data approach, where data is accessible through both Kafka APIs and as Iceberg tables for analytics workloads.

   10