ENTP: ENCODER-ONLY NEXT TOKEN PREDICTION

Summary from the TLDR AI newsletter

This paper shows a novel design architecture for using encoder-only transformers as next token prediction models. This has been done in other ways before, but the researchers find a great way to prevent cheating. The models have hints of strong performance at scale and also are capable of in-context learning.