This post explains how to build a modern data lakehouse using DuckDB and MinIO. It describes the installation of the HTTPFS extension in DuckDB to connect to MinIO, an object storage solution. The process includes setting up new connections in Airflow and modifying scripts to replace local disk storage with MinIO, enhancing the pipeline performance and reliability.
Table of contents
Install httpfs in DuckDBAdd new connection “minio” in Airflow connection menu.Add new function in file function/general.pyChange file brz_patientsChange file slv_patientsFile stored to MinIO ServerSort: