WHY DO LEARNING RATES TRANSFER?

Summary from the TLDR AI newsletter

A theoretical paper that attempts to explain the success of MuP hyperparameter transfer. Its authors find that the largest eigenvalue of the training loss Hessian is independent of the width and depth of the network.