SOTA ON SWEBENCH-VERIFIED: (RE)LEARNING THE BITTER LESSON

Summary from the TLDR Web Dev newsletter

CodeStory achieved state-of-the-art results on the swebench-verified benchmark by scaling its AI agent's test-time inference, achieving a 62.2% success rate. This success contradicted the assumptions of its initial “smart” approach, which used a Monte Carlo Tree Search (MCTS) framework. Massive scaling with multiple VMs and Anthropic accounts was much better, showing the power of "scale over cleverness."