SPQR: A TECHNIQUE FOR HIGH-EFFICIENCY LLM COMPRESSION

Summary from the TLDR AI newsletter

The paper presents Sparse-Quantized Representation (SpQR), a new format and technique that allows almost lossless compression of large language models (LLMs), overcoming the usual accuracy losses from quantization. SpQR enables powerful LLMs to run on common devices like laptops and mobile phones without performance degradation, offering over 4x memory compression gains and faster inference than traditional methods.