ON-POLICY DISTILLATION

Summary from the TLDR AI newsletter

Thinking Machines Lab demonstrated that training smaller AI models by having them learn from their own mistakes, by grading them with a larger teacher model, achieves the same reasoning performance as RL at 9-30x lower cost. "On-policy distillation" combines the relevance of learning from your own outputs with the dense feedback of traditional distillation, reaching 70% on AIME'24 math problems in just 150 training steps compared to 17,920 GPU hours of RL.