A NEW COMPLEX BENCHMARK FOR AI SYSTEMS

Summary from the TLDR AI newsletter

A new dataset called GPQA presents a real challenge with 448 tough multiple-choice questions in biology, physics, and chemistry. Even domain experts struggle, scoring around 65% accuracy, while non-experts only manage 34%. Advanced AI systems like GPT-4 achieve just 39% accuracy. This dataset is aimed at developing methods for supervising AI outputs in complex scientific questions.