REDUCING PPO MEMORY USAGE IN RLHF

Summary from the TLDR AI newsletter

PPO is a pain in RLHF because you need to have 3 models floating around. However, they all stay close and only require small updates from one another. Enter LoRA. If you instead use adapters you can dramatically reduce memory costs without losing performance. Beautiful idea in its simplicity.