SEQUENTIAL A/B TESTING KEEPS THE WORLD STREAMING NETFLIX PART 1: CONTINUOUS DATA

Summary from the TLDR Web Dev newsletter

Netflix developed a statistical procedure for A/B testing to detect changes in software performance. The traditional fixed-horizon testing methods are not suitable for this task because they require waiting for a predetermined amount of data before analysis, which can delay the detection of important regressions. The new method, called sequential testing, allows for continuous analysis of data streams and provides guarantees on false positive rates. It has been successfully implemented at Netflix to prevent bugs and performance regressions from reaching production.