SPARK TIPS. PARTITION TUNING

Summary from the TLDR Data newsletter

Spark partition sizing is a throughput and stability lever: aim for enough partitions to use cores without creating tiny tasks, filter early, avoid unnecessary shuffles, and repartition around joins or writes when the current layout is wrong. The practical advice covers AQE, skew, spills, checkpoint barriers, and small-file tradeoffs.