WEIGHT AVERAGING OF REWARD MODELS

Summary from the TLDR AI newsletter

Reward models are used in RLHF to represent human preference, although the model being aligned often “hacks the reward” and achieves unfavorable performance. By merging multiple reward models, which remain linearly mode connected, the resulting aligned model is preferred 79% of the time over one aligned with a single reward model. Model merging is strange, and may just be regularization, but it has worked surprisingly well in general models and now has been shown to work as a training step for the general language model pipeline.