A DECISION-ONLY JUDGE MATCHES GPT-6 ON ROUTINE EVALS FOR 0.36% OF THE FEE

Summary from the TLDR Dev newsletter

CMU tested Jev, a judge that returns only a verdict and label probabilities, against 16 LLM and reward-model judges. It comes within three points of GPT-6 on preference and factuality at 277 times lower fees, though it fails on difficult correctness tasks.