YOU COULD'VE INVENTED TRANSFORMERS

Summary from the TLDR AI newsletter

The fundamental architecture of LLMs can be explained as a series of straightforward steps, starting from n-grams' 0-count problem through embeddings, neural LMs, and self-attention. Transformers are sophisticated, but obvious in retrospect—heavily tweaked MLPs that solve RNNs' information propagation problems.