SCHEDULE FREE OPTIMIZER

Summary from the TLDR AI newsletter

A few weeks ago, a new optimizer from Meta made the rounds as a promising alternative to Adam. This paper outlines the method in more detail, including the online updates portion of the method. In general, it seems like a promising result, especially when the total number of desired training steps is unknown at the beginning of training.