THE PROBLEM WITH REASONERS

Summary from the TLDR newsletter

o1 reasoners are the most exciting models to come out since the original GPT-4, but they suck on important problems. o1 uses a reinforcement learning (RL) environment where reasoning steps are actions, previous tokens are observations, and reward is the solution's correctness. RL is great for games, high-frequency trading, protein folding, and other problems, but not for open-ended thought without clear feedback. RL-based reasoners struggle to perform in domains outside of their training, which is most fields. Transformative AI will likely come from scaling rather than through reasoning models.