SELF-PLAY FINE-TUNING CONVERTS WEAK LANGUAGE MODELS TO STRONG LANGUAGE MODELS

Summary from the TLDR AI newsletter

One of the big drawbacks of current language models is that you need extremely expensive human preference data to make them work well. A big research direction recently has been figuring out if we can have the language models perform self-play to improve without collecting this data. SPIN is a new method that takes a step in that direction with just SFT data - it can dramatically improve a base model’s performance across various tasks.