HOW ENABLING TWO SETTINGS TRIPLED OUR SCORES ON THE ARC-AGI-3 BENCHMARK

Summary from the TLDR AI newsletter

GPT-5.6 Sol scores just 7.8% on the ARC-AGI-3 benchmark despite having solved longstanding open problems in mathematics and beaten games like Pokémon FireRed. Benchmarks rarely measure AI models in isolation. They also measure less visible choices about API settings, harness design, and prompting. Researchers discovered that turning on retained reasoning and compaction in ChatGPT and Codex tripled scores and cut output tokens by 6x on the benchmark.