QUANTIZATION FROM THE GROUND UP

Summary from the TLDR AI newsletter

Quantized models are actually pretty good. A 16-bit to 8-bit quantization carries almost no quality penalty - the difference with a 4-bit quantization is more noticeable, but it would only perform about 90% as well as the original. It's worth experimenting with these models as they're much smaller and can run on more systems. This article explains how model parameters work, what quantization is, how quantization is applied in practice, and the effects of quantization on model accuracy.