THE UNIT-SCALED MAXIMAL UPDATE PARAMETRIZATION

Summary from the TLDR AI newsletter

muP is a way to ensure that a model's hyperparameters are independent of the model's size. This work also ensures that you can transfer across quantized models.