SMEVALS

Summary from the TLDR AI newsletter

smevals is a framework for running evals against AI models, small and large. An Eval is a collection of Tasks used to determine how good a particular model or model-and-harness configuration is at a specific high-level capability. Tasks are the individual exercises that a model must complete for its abilities to be evaluated. Evals can optionally be grouped into Suites of related Evals, primarily as a mechanism for organizing them on disk.