BRINGING DUCKLAKE TO DATAFUSION

Summary from the TLDR Data newsletter

DuckLake is now implemented for Apache DataFusion as a production-ready, open-source catalog backend, giving Hotdata transactional metadata, snapshots, and schema management for Parquet tables in object storage. The design supports multiple relational catalog backends and logical catalogs, enabling millions of isolated, ephemeral databases for AI agents without duplicating infrastructure. TPC-H benchmarks across 22 queries and three dataset sizes showed no measurable overhead versus direct Parquet access, with slightly better performance due to less metadata discovery during query planning.