EMBARRASSINGLY SIMPLE SELF-DISTILLATION IMPROVES CODE GENERATION

Summary from the TLDR AI newsletter

Simple self-distillation (SSD) is a process where models are fine-tuned on samples of their output with standard supervised fine-tuning. It offers a complementary post-training direction for improving LLM code generation. SSD can improve code generation in models using only raw outputs. This study looks at why such a simple method can work.