WHAT ACTUALLY KEEPS AN AI BENCHMARK USEFUL? SCALE

Summary from the TLDR Dev newsletter

A study of 60 widely used LLM benchmarks shows that nearly half have reached saturation, making them unable to differentiate top models due to measurement noise. Some factors contributing to saturation include benchmark age and test set size, while commonly assumed safeguards like private test sets and output formats do not effectively prevent this issue.