PASS@K IS MOSTLY BUNK

Summary from the TLDR AI newsletter

Pass@k is the probability that at least one of k different attempts will succeed. It is one of the most common metrics used for agents. The problem with the metric is that it's exponentially forgiving, and humans interacting with agents aren't nearly that forgiving. Pass@k should be a metric that's rarely used, and carefully justified every time it is used.