THE FICKLENESS OF SCALING LAWS

Summary from the TLDR newsletter

Scaling laws have defined tech in the last five years. Large language models have demonstrated consistent returns to scaling. However, these 'laws' apply to very few other fields. Little work has been done to systematically understand, in theory or in general, what causes scaling laws or determines their slope and robustness. This article looks at what causes scaling laws, what their preconditions are, when they stop, why they have worked in some fields but not others, and more.