NEW BENCHMARK FOR CODE GENERATION

Summary from the TLDR AI newsletter

TACO is a new benchmark for assessing a system’s ability to generate code. It is substantially larger than existing datasets and contains more challenging problems. On easy problems, GPT-4 gets 30% correct, while it only achieves 2% on the hardest subset of questions.