LORA LEARNS LESS AND FORGETS LESS

Summary from the TLDR AI newsletter

LoRA is a popular way to fine-tune models to inject style or knowledge. This paper explores the trade off between power and forgetting when using LoRAs. The research finds that LoRAs are able to learn less than full fine tuning, but maintain more of the original “out of distribution” performance.