FREE PROCESS REWARDS WITHOUT PROCESS LABELS

Summary from the TLDR AI newsletter

Process reward models (PRMs) offer finer-grained feedback by evaluating reasoning trajectories step-by-step, unlike outcome reward models (ORMs), which assess entire responses. However, training PRMs requires detailed intermediate annotations, posing significant challenges. This paper shows that an implicit PRM can be derived at no additional cost by training an ORM on response-level labels using log-likelihood ratios of policy and reference models, enabling optimization without requiring specific loss objectives.