Now that you can use <a href="https://duckdb.org/" target="_blank" rel="noopener nofollow">DuckDB</a> to power your data lakehouse through <a href="https://ducklake.select/" target="_blank" rel="noopener nofollow">DuckLake</a>, you’ll also save space on snapshots due to the ability to reference parts of parquet files (yes, you can keep all old versions, with little impact to storage), and you’ll get improved performance for small change operations due to data inlining, which lets data be stored directly within the metadata database (<a href="https://www.sqlite.org/" target="_blank" rel="noopener nofollow">SQLite</a>, <a href="https://www.postgresql.org/" target="_blank" rel="noopener nofollow">PostgreSQL</a>, etc.).
With a little help from <a href="https://docs.getdbt.com/" target="_blank" rel="noopener nofollow">dbt</a> and an <a href="https://github.com/duckdb/dbt-duckdb/issues/564" target="_blank" rel="noopener nofollow">unreleased branch</a> of <a href="https://github.com/duckdb/dbt-duckdb" target="_blank" rel="noopener nofollow">dbt-duckdb</a> adapter, I was able to design a data lakehouse strategy, covering data ingestion, transformation, and exporting, almost exclusively based on SQL, on top of DuckDB and the newly released DuckLake!
The code is available as open source on GitHub, at <a href="https://github.com/DataLabTechTV/datalab" target="_blank" rel="noopener nofollow">DataLabTechTV/datalab</a>, and the <code>README</code> will cover most of the details you need to understand it and get it running. I wrote a <a href="https://datalabtechtv.com/posts/data-lakehouse-dbt-ducklake/" target="_blank" rel="noopener nofollow">blog post</a> where I covered some of the most interesting components or issues, and provided a few comments about the whole process. You can also see data lab in action, and learn more about it, by watching the <a href="https://youtu.be/zn69Q7FiFfo?si=WBwDJYKPQx-_zTIa" target="_blank" rel="noopener nofollow">video</a> that I just uploaded about it this week!
Also, thanks for all the good feedback I got from this squad, with my last videos about PostgreSQL Maximalism. I was quite happy you guys enjoyed it!

Your go-to source for the latest in database technology, trends, and tips. Covering Postgres, MySQL, Kafka, Data Warehouses, and more. 

Database Daily

A data lakehouse implementation using DuckDB and DuckLake offers improved storage efficiency through parquet file referencing and better performance for small changes via data inlining. The solution combines dbt for transformations with an unreleased dbt-duckdb adapter to create an SQL-based data pipeline covering ingestion, transformation, and export. The approach leverages DuckLake's ability to store metadata in databases like SQLite or PostgreSQL while maintaining version history with minimal storage impact.

Data Lakehouse with dbt and DuckLake