unknown

DuckDB and Polars were benchmarked on datasets up to 2 TB to compare performance and memory usage under extreme conditions. DuckDB maintained peak memory below 2.5 GB across all tests while Polars consumed up to 17 GB on large single files due to mmap-based I/O, though this dropped to 4.3 GB with partitioned data. Polars with forced async mode achieved DuckDB-like memory efficiency (750 MB) but sacrificed execution speed. File layout proved critical: partitioning the same 140 GB dataset into smaller files reduced DuckDB's memory by 8× and Polars' by 4×. Row group size optimization improved performance by 2.8× for Polars and 1.6× for DuckDB. The choice between engines depends on memory constraints and data organization, with DuckDB favoring memory-constrained environments and Polars excelling with well-partitioned data on memory-abundant systems.

DuckDB vs. Polars: Performance & Memory on Parquet Data

Keep up with the latest content around DuckDB - curated by veteran Data engineers and DuckDB daily users!