CONVEX OPTIMIZATION THEORY AND LEARNING-RATE SCHEDULING FOR LARGE MODEL TRAINING

Summary from the TLDR AI newsletter

Learning-rate schedules for large models closely match theoretical bounds from non-smooth convex optimization. These authors provide a bound for constant schedules with linear cooldown, showing cooldown's practical benefits through the absence of logarithmic terms in the bound. Their findings enabled practical improvements in training Llama-type models through optimal learning-rate extension and cross-schedule transfer.