APACHE SPARK RESOURCE OPTIMIZATION: LESSONS FROM EIGHT MILLION JOBS A DAY

Summary from the TLDR Data newsletter

Salesforce runs about 8 million Spark jobs a day and found some workloads left executors idle more than 90% of the time. Sizing targeted workloads to measured usage reduced weekly compute costs by 31%, while runtime, failures, and retries remained within acceptable levels.