FINE-TUNING LLMS TO 1.58BIT: EXTREME QUANTIZATION MADE EASY

Summary from the TLDR AI newsletter

BitNet, developed by Microsoft Research, introduces a transformer architecture that reduces LLM computational and memory requirements by using ternary precision (-1, 0, 1) equating to 1.58 bits per parameter. Models are required to be trained from scratch. BitNet can also fine-tune existing models to this low-precision format, maintaining strong performance on downstream tasks. This approach significantly reduces energy consumption and improves inference speed using specialized kernels for efficient matrix multiplication.