TRUSTWORTHY A/B PATTERNS AND THE WINNER'S CURSE: LESSONS FROM SEVEN LARGE-SCALE REPLICATIONS

Summary from the TLDR Marketing newsletter

Large-scale replications show that popular A/B test patterns often produce much smaller gains than earlier reports suggest, and most reported large lifts come from underpowered studies. Even very large experiments often lack enough power for core metrics like revenue-per-user and purchase rate, so teams often must rely on surrogate metrics like click rate and add-to-cart rate. Real-world results from companies such as Bing, Amazon, and Talabat show that typical trustworthy lifts are often below 1%, not the double-digit gains seen in case libraries. The evidence supports that low-power tests below 50% power create exaggerated winners and frequent false positives, which explains why many published pattern wins fail to replicate at scale.