ARE YOUR LLMS CAPABLE OF STABLE REASONING?

Summary from the TLDR AI newsletter

Reasoning is a huge opportunity for models, especially for critical applications like real world deployment. However, benchmarks that are meant to test reasoning ability don't show how stable a model is across novel tasks. This paper introduces a new benchmark called G-Pass@k which evaluates a model's peak and stable reasoning ability.