SCHEDULE FREE OPTIMIZATION

Summary from the TLDR AI newsletter

Researchers from Meta have been teasing a new optimizer on X. They have now released the code along with various integrations. The optimizer has no LR schedule, which means you don’t need to know the full number of training steps beforehand. It has been shown empirically to work on a wide variety of problems including language models.