ABC INVARIANCE

Summary from the TLDR AI newsletter

muP is a way to tune your hyperparameters to transfer from smaller to larger models. This GitHub gist shows a great theorem in practice which states that you can change where you scale model outputs and it doesn't affect final transfer performance.