SCALING LAWS AND COMPUTE-OPTIMAL TRAINING BEYOND FIXED TRAINING DURATIONS

Summary from the TLDR AI newsletter

The original scaling laws papers used a Cosine learning rate schedule which doesn't achieve optimal loss if the Cosine period isn't aligned with the total training steps. This makes it challenging to train enough models to create meaningful scaling laws. This paper introduces the idea of a constant learning rate with a cool down as a way to reduce GPU costs for scaling law creation.