REWARD MODELS CAN BE TOO SENSITIVE

Summary from the TLDR AI newsletter

Meta studied how reward models can overreact to equally good responses, leading reinforcement learning toward reward hacking. The paper proposes measuring both discriminative ability and specificity, then using Monte Carlo dropout to cluster rewards into safer discrete signals.