HOW WE EXTENDED APACHE DATAFUSION TO EXECUTE ONE QUERY ACROSS MANY MACHINES

Summary from the TLDR Data newsletter

Datadog built Distributed DataFusion to let a single Apache DataFusion query run across multiple machines, giving heavy interactive queries much more compute while keeping lightweight queries on one machine to avoid unnecessary overhead. In production, some expensive queries ran up to 10× faster at the same total cost. Datadog released the system as an open-source framework that teams can adapt to their own data sources and infrastructure.