I'M NOW CONVINCED SINGLE-NODE DUCKDB IS THE BEST ENGINE AT ANY REALISTIC SCALE (REDDIT THREAD)

Summary from the TLDR Data newsletter

A Databricks user found that Spark, Trino, and DuckDB all spent most of their time waiting on S3, with CPUs about 90% idle. After running four times as many read threads as CPU cores, one DuckDB machine beat Trino on the same hardware by 1.7 to 2.1 times on 10 TB queries. Against a Trino cluster with 5 to 13 nodes, it was 2 to 3 times slower but about half the cost.