TLDR Index

TLDR AI · ENGINEERING & RESOURCES

TORCH COMPATIBLE MUON OPTIMIZER

github.comGitHub repository

Summary from the TLDR AI newsletter

Muon was the optimizer used for the GPT-2 training record. It is a momentum adapted SGD style method. This repository contains an implementation that can be dropped in for AdamW.

Read the original article ↗View the archived TLDR AI issue