WHEN LLM JUDGES AGREE, SHOULD WE BELIEVE THEM?

Summary from the TLDR Dev newsletter

Agreement among LLM judges can overstate confidence when models share prompts, architectures, or blind spots. A dependence-aware aggregation method models both judge reliability and pairwise correlations, improving accuracy by 9% to 14% over weighted majority voting across three evaluation tasks.