LORA WITHOUT REGRET - THINKING MACHINES LAB

Summary from the TLDR AI newsletter

LoRA, the most popular method for fine-tuning AI models, can match the performance of expensive full fine-tuning when applied correctly, delivering the same results at two-thirds the cost. The key is applying LoRA to all network layers (especially MLPs rather than just attention layers) and ensuring the rank is high enough for the dataset. Thinking Machines found ranks of 256-512 work well for typical post-training scenarios combined with 10× higher learning rates than fine-tuning.