EXAMINING LLM PERFORMANCE ON PUBLIC BENCHMARKS

Summary from the TLDR AI newsletter

How overfit are popular LLMs on public benchmarks? New research from Scale AI SEAL revealed that Mistral & Phi are overfitting benchmarks, while GPT, Claude, Gemini, and Llama are not. The researchers produced a new eval GSM1k and evaluated public LLMs for overfitting on GSM8k.