650GB of Data (Delta Lake on S3). Polars vs DuckDB vs Daft vs Spark.

A performance comparison testing DuckDB, Polars, Daft, and Spark on a 650GB Delta Lake dataset stored in S3, using a single 32GB EC2 instance. DuckDB completed the aggregation query in 16 minutes, Polars in 12 minutes, Daft in 50 minutes, and PySpark in over an hour. The experiment demonstrates that single-node data processing frameworks can effectively handle large lakehouse datasets without requiring expensive distributed clusters, challenging the assumption that distributed computing is necessary for most data workloads.