NOTES ON PRETRAINING PARALLELISMS AND FAILED TRAINING RUNS

Summary from the TLDR AI newsletter

Pretraining runs often fail. This article looks at all the ways that things can go wrong and why training is such a precarious operation. The key culprits seem to be breaking causality and adding bias.