PHYSICS OF LANGUAGE MODELS: PART 3.3

Summary from the TLDR AI newsletter

The 3rd part of a series, this paper explores scaling laws and how many bits are required to store knowledge in a model. The answer seems to be around 2 bits of knowledge per parameter.