MUP FOR SPARSE MODELS

Summary from the TLDR AI newsletter

muP is a powerful tool used by all frontier model labs to transfer hyper parameters tuned on small models to larger, more expensive training runs. This paper explores how to do that for sparse models, thus dramatically improving training performance while reducing compute costs.