KV QUANT - SCALING TO 10M TOKENS

Summary from the TLDR AI newsletter

Quantizing the KV cache, which is a technical detail of the Transformer architecture, allows it to use less memory at inference time. Quantization is the shrinking of the floating point precision with minimal loss in quality.