HOW TO USE DECISION MODELS FOR EVALS

Summary from the TLDR newsletter

Decision models are an appealing alternative to LLM judges for automated evals. However, whether they're the right choice depends on the cost, performance, and latency requirements of the use case. Many people leave the default decision threshold at 0.50 without checking whether that's appropriate for the task, leaving a lot of performance on the table. This post shows how to tune thresholds to get the best results from decision models, compares cost and performance between models, and explores how to interpret model scores and how calibration can make them more useful.