NO, GPT4 CAN’T ACE MIT

Summary from the TLDR Web Dev newsletter

A paper seemingly demonstrating that GPT-4 could ace the MIT EECS + Math curriculum recently went viral on Twitter, getting over 500 retweets in a single day. However, three MIT EECS seniors took a closer look at the paper and found enough problems with it to debunk it. For example, even though the authors of the paper said they manually reviewed the published dataset for quality, the three MIT seniors found clear signs that a significant portion of the evaluation dataset was contaminated in such a way that let the model cheat like a student who was fed the answers to a test right before taking it.