CAN WE TRUST LLMS TO JUDGE WHETHER A RESEARCH IDEA IS NOVEL?

Summary from the TLDR Dev newsletter

Ai2's benchmark, built from ICLR 2026 peer reviews, tests whether LLMs can judge research novelty and finds that small prompt changes can swing accuracy by more than 50 points on AI-generated ideas. Extra retrieval and reasoning did not reliably help, so teams using LLM judges should test their stability on the kinds of ideas they plan to score.