GZIP PREDICTS DATA-DEPENDENT SCALING LAWS

Summary from the TLDR AI newsletter

Scaling laws are a way to predict how well models will perform at certain sizes with a certain amount of data. They are expensive to obtain. This paper explores using gzip compression ratio as a strong signal to predict a data-dependent scaling law.