RUNNING APACHE SPARK EXPERIMENTS IN MY SLEEP (AND ON A PLANE)

Summary from the TLDR Data newsletter

Netflix's member-sessionizer pipeline turns tens of billions of hourly client events into viewing sessions, but scale bottlenecks made ordinary tuning painfully slow. Instead of optimizing one Spark run at a time, the team built a parallel experiment loop that launched, monitored, and compared multiple production-scale tests to expose memory and shuffle bottlenecks faster.