BENCHMARKS ARE MORE BROKEN THAN WE COULD HAVE IMAGINED

Summary from the TLDR Data newsletter

Horizon audited over 5,000 AI benchmark tasks across 20 datasets and confirmed 29 were broken, with problems including leaked answers, incomplete tests, gameable graders, and failing reference solutions. Many of these flaws actually inflated model scores, highlighting how benchmark results can overstate capabilities when the underlying tasks and graders are unreliable.