RL IS EVEN MORE INFORMATION INEFFICIENT THAN YOU THOUGHT

Summary from the TLDR AI newsletter

It takes way more FLOPs to get a single sample in RL than it does in supervised learning. RL requires unrolling a whole thinking trajectory tens of thousands of tokens long to get a single reward signal at the end. In pretraining, you get a signal on every single token you train on. For most of training, the information density per sample is way lower for RL compared to supervised learning.