INITIALIZE A SMALL MODEL FROM A BIG ONE

Summary from the TLDR AI newsletter

Lots of work has been done in distillation and initialization such as the lottery ticket hypothesis. This work gives a simple mechanism to take a much larger model and initialize a smaller one from it. It dramatically improves smaller model performance.