THE BENCHMARKING CRISIS

Summary from the TLDR Data newsletter

AI code generation is helping database engines game ClickBench. One, pgrust, groups strings by hash value alone, caches predicates the benchmark repeats, and on one query skips most of the data. Any benchmark stops being useful once systems optimise for it, and vibe-coded engines have no customers to keep them honest.