AUDITING THE RELIABILITY OF CODING BENCHMARKS

Summary from the TLDR AI newsletter

OpenAI examined SWE-Bench Pro's construction, model failures, and task metadata, concluding that roughly 30% of its public tasks were broken. The analysis showed how flawed evaluations could distort assessments of coding ability, safety, and model progress.