TINYBENCHMARKS: EVALUATING LLMS WITH FEWER EXAMPLES

Summary from the TLDR AI newsletter

Evaluating language models is typically done with hand-curated benchmarks. Some of these benchmarks are quite large, with some at over 14k examples, which leads to high expense and noise in evaluation. This work shows that you can reliably evaluate language model performance with as few as 100 examples from popular benchmarks.