DO YOU EVEN BENCHMARK? HOW WE EVALUATE AGENTS AT SCALE

Summary from the TLDR Data newsletter

PostHog tests AI agents on real tasks, such as finding the bugs behind 33 past incidents, where its review agent caught 62%, or checking whether playback failures affected real users. It isolates each test so agents cannot copy earlier answers and is building resettable environments with fixed data to make comparisons fair.