30% OF "HUMANITY'S LAST EXAM" ANSWERS ARE LIKELY WRONG

Summary from the TLDR AI newsletter

FutureHouse researchers found that nearly a third of biology and chemistry questions in the prominent PhD-level AI benchmark contain answers directly contradicted by peer-reviewed literature. The errors stem from HLE's unusual design that created "gotcha" puzzles that even confused human experts.