BAD BENCHMARKS AND EVALS: SENIOR SWE-BENCH, NAPKIN MATH, AND WINTER TIRES

Summary from the TLDR AI newsletter

Evals are more about avoiding mistakes than following some particular process. There isn't any step-by-step guide that will work, but there are approaches to experimental design that can teach you how to do better. This post aims to help people become better at benchmarking and evals. It looks at three different kinds of benchmarks: one for baseline numbers for performance 'napkin math' estimates, one set of AI model evals, and one on car tires.