DYNAMIC 4-BIT QUANTIZATION

Summary from the TLDR AI newsletter

The Unsloth team aims to shrink a 20GB language model into 5GB without sacrificing accuracy. There are many algorithms that attempt this, but there are a number of challenges around outliers and compressibility. Llama is a famously hard model to quantize. The method selectively avoids quantizing certain parameters, which greatly improves overall accuracy.