THE PRACTITIONER'S GUIDE TO THE MAXIMAL UPDATE PARAMETERIZATION

Summary from the TLDR AI newsletter

Maximal Update Parameterization, or muP, is a way to initialize your model so that you can transfer hyperparameters at any scale. This blog from Eleuther and Cerebras includes a minimal nanoGPT example and lots of instruction on how the process works.