THE EXTREME INEFFICIENCY OF RL FOR FRONTIER MODELS

Summary from the TLDR AI newsletter

A key difference between pre-training and reinforcement learning (RL) is their information efficiency. Pre-training via next-token-prediction provides models with a token worth of information to learn from for every token the model produces during training, while RL requires a long chain of thousands and even millions of tokens before revealing to the model a single bit of information. RL provides models with much less information to learn from per GPU-hour. This has serious implications for how much the RL paradigm can achieve.