ACCELERATING LLM INFERENCE WITH EFFICIENT QUANTIZATION

Summary from the TLDR AI newsletter

QoQ, a novel quantization algorithm, accelerates large language model inference by using 4-bit weights, 8-bit activations, and 4-bit KV cache.