MODEL SIZE SCALING IN 2023-2031

Summary from the TLDR AI newsletter

Targeting a particular speed of token generation puts a constraint on the total parameters of the model. If there isn't enough pretraining compute, models will remain smaller. This article looks at these considerations and estimates model sizes feasible for each year between 2023 and 2031. There are many assumptions that go into the estimates, which predict total parameters for models to reach 1.4 quadrillion in 2031.