BERTS ARE IN-CONTEXT GENERATIVE LEARNERS

Summary from the TLDR AI newsletter

In an alternative world, BERT models would have been discovered to be in-context learners instead of their decoder-only GPT counterparts. This paper explores when that is true and finds that, unsurprisingly, BERTs are excellent at information retrieval but struggle with knowledge acquisition - likely due to the bidirectional attention mechanism.