DERIVING MUON

Summary from the TLDR AI newsletter

Adam has been the dominant optimizer for years in deep learning. However, recently the community has found that Muon might be a viable alternative. It accomplishes many of the same things as muP without requiring modifications to the model. This post describes some of the theoretical motivations behind the optimizer.