SIMPLER RL ALGORITHMS BEAT PPO

Summary from the TLDR AI newsletter

REINFORCE is a simple, standard, and easily understood RL method. It is hard to train stably when used in simulators. PPO is much more performant and stable in general. Gemini uses REINFORCE and GPT-4 is believed to use PPO.