SYNTHETIC DATA AND NEXT LEVEL CURATION GIVE BIG HUMANEVAL BOOST

Summary from the TLDR AI newsletter

Textbooks may be all you need for extreme AI performance. By using a highly curated set of data and synthetic textbooks with exercises, a 1.3B parameter model achieves 50+ on HumanEval, a challenging code benchmark. For context, this rivals ChatGPT and far surpasses the original 12B Codex model. This paper follows the Tiny Stories line of work and has lots of nice tidbits.