USE SMALL MODELS TO STUDY LARGE MODELS

Summary from the TLDR AI newsletter

Instabilities in training at large scales are hard to reproduce for most researchers. This paper shows how to reproduce instabilities in small models and verifies that many commonly reported fixes work just as well on these small models.