QUEST: STABLE TRAINING OF LLMS WITH 1-BIT WEIGHTS AND ACTIVATIONS

Summary from the TLDR AI newsletter

QuEST allows stable training with 1-bit weights and activations. QuEST achieves this by improving two key aspects of QAT methods: accurate and fast quantization of the (continuous) distributions of weights and activations via Hadamard normalization and MSE-optimal fitting, and a new trust gradient estimator based on the idea of explicitly minimizing the error between the noisy gradient computed over quantized states and the "true" (but unknown) full-precision gradient.