ADEMAMIX OPTIMIZER

Summary from the TLDR AI newsletter

This paper from Apple proposes an alternative to traditional exponential moving average optimization that uses contributions from older gradients to dramatically improve learning convergence.