AI BENCHMARKS DON'T MEAN WHAT YOU THINK THEY MEAN

Summary from the TLDR AI newsletter

Benchmarks are trotted out whenever a new model is released, but what do they actually measure? ngrok's Sam Rose dug into the papers, code, and critiques to understand what 14 popular AI benchmarks actually mean [12]. E.g., SWE-bench Verified = fixing small bugs in 12 popular