PRE-TRAINING UNDER INFINITE COMPUTE

Summary from the TLDR AI newsletter

Compute for pretraining grows 4× annually, but web data increases just 1.03× per year, forcing a shift toward more efficient learning algorithms. Regularization parameters 30× higher than standard practice prevent overfitting when models see the same data repeatedly. Ensembling independently trained models achieves lower loss than simply making models larger. The combination cuts data requirements by 5× to match baseline performance and directly translates to improvements on pretraining benchmarks.