BartWullems's blog is a platform for sharing insights and expertise in Microsoft technologies, software development best practices, and programming tutorials. Through articles, code samples, and technical discussions, BartWullems offers insights into building scalable, maintainable, and performant software applications using Microsoft technologies such as .NET, Azure, and Visual Studio. Readers can learn about design patterns, development techniques, and tooling recommendations to enhance their skills and productivity as Microsoft developers. Additionally, BartWullems provides updates on the latest developments in Microsoft ecosystem, community events, and industry trends to help developers stay informed and engaged in the Microsoft developer community.

The Art of Simplicity

DuckDB is an in-process analytical database optimized for OLAP workloads that can query local Parquet and Delta Lake files directly without loading them fully into memory. Unlike pandas, DuckDB uses columnar vectorized execution and predicate pushdown to read only the necessary row groups and columns, making it significantly faster and more memory-efficient for large files. The post covers installation, querying single and multiple Parquet files via glob patterns, using delta_scan for correct Delta table reads, creating persistent connections with named views for multi-table notebook workflows, and mixing DuckDB with existing pandas DataFrames. A decision table clarifies when DuckDB is preferable over pandas or delta-rs.

Run SQL queries on local parquet and delta files using DuckDB

Using a persistent connection for Multiple Queries