SMEVALS - A SMALL EVAL SUITE FOR EVALUATING MODELS, PROMPTS, AND HARNESSES

Summary from the TLDR Data newsletter

Smevals is a lightweight Python CLI for running task-specific evals across multiple models to find the cheapest one that meets a quality bar. You define tasks, write YAML graders with custom checker scripts, run them against a set of model configs, and view scored results in a local dashboard or static site.