FIRST-PRINCIPLES ON AI SCALING

Summary from the TLDR newsletter

Scaling laws indicate that Large Language Models (LLMs) will get substantially better at predicting words with the addition of more data and computational power. However, while it might be feasible to increase dataset size, there are potential barriers to making them more than 10 times larger than their current state, such as cost. Scaling LLMs to maximum potential would cost substantially more than the GDP of the planet with current technology. There is likely a computational limitation. Current scaling laws could be inaccurate due to the small amount of data and computation used to fit the model.