WE TESTED JEV ON 100 REAL AGENT CALLS. HOW EASY IS IT TO BEAT A CONSTANT?

Summary from the TLDR Dev newsletter

Tests on 100 production agent tool calls found that 79% were benign, so a classifier had to beat a constant benign baseline before its accuracy was useful. Jev's errors, class imbalance, and benchmark design show why evaluations need deployment-aware baselines and why a strong-looking score can still lose to simple code.