SINQ

Summary from the TLDR AI newsletter

SINQ (Sinkhorn-Normalized Quantization) is a novel, fast, and high-quality quantization method designed to make any large language models smaller while keeping their accuracy almost intact. It uses dual scaling to make models less vulnerable to outliers. SINQ's Sinkhorn-normalized optimization, which iteratively rescales rows and columns to balance their variance, makes errors more spread out and less severe, preserving model accuracy even at 3-bit precision. Weights become inherently easier to quantize when the overall matrix imbalance is reduced, leading to more stable behavior across layers and consistently higher accuracy even at very low bit-widths.