HOW TO AVOID WILL ROGERS' PHENOMENON IN A/B TESTING

Summary from the TLDR Marketing newsletter

Will Rogers' Phenomenon occurs when moving people between groups makes the average performance of both groups improve, even though no one actually improved. In A/B testing, this can happen when users cross a threshold during an experiment and get reclassified from casual to power users, which can make both segments appear to perform better. This can create false conclusions about a feature's impact and mislead marketing or product decisions. Freeze segment definitions before the experiment starts and track segment migration separately to avoid the problem.