REVISITING HARD QUESTIONS WITH REPLAY BUFFERS

Summary from the TLDR AI newsletter

ZPPO stores difficult questions in a replay buffer so the model can repeatedly train on them rather than seeing them only once. The method is designed to strengthen learning on challenging examples and improve rollout accuracy.