CODEMMLU

Summary from the TLDR AI newsletter

Language models are getting so good at writing code that many benchmarks are saturating. This paper suggests a benchmark that is harder and aims to determine how models perform on reasoning and code generation tasks.