SCALING OPTIMAL LR ACROSS TOKEN HORIZON

Summary from the TLDR AI newsletter

This paper explores how to scale learning rate when you increase the amount of training data a model sees. Llama did exponential -0.28 and this paper suggests exponential -0.33.