WHEN DO WE NEED ON-POLICY DISTILLATION? DISTILLING ON OFFLINE STUDENT ROLLOUTS IS OFTEN BETTER

Summary from the TLDR AI newsletter

On-policy distillation has become increasingly popular for transferring teacher capabilities to student models. Semi-OPD, which distills from offline rollouts generated by the initial student, can often outperform OPD in both accuracy and training efficiency. The choice between OPD and Semi-OPD depends on the alignment between the initial teacher and student. OPD is beneficial only when the two are highly aligned with high overlap ratios.