ANCHORED PREFERENCE OPTIMIZATION

Summary from the TLDR AI newsletter

Tuning models to align with human preferences usually requires some post training. However, when training these models it is not clear why one example is better than another. APO is a way for models to degrade existing examples and then use that degraded example to anchor the difference in preference.