INTRIGUING PROPERTIES OF QUANTIZATION AT SCALE

Summary from the TLDR AI newsletter

We’ve talked about precision before, that’s how much space it takes to store a number. And how many unique values that number can represent. When you reduce the precision, AI models take less space on the computer, but weird things start to happen because you represent fewer numbers for the parameters. This paper studies what happens when you quantize before training at scale. They start to poke at the magical 6B parameter performance barrier.