MECHANICS OF NEXT TOKEN PREDICTION WITH SELF-ATTENTION

Summary from the TLDR AI newsletter

Next token prediction is a simple objective that leads to complex behaviors. This work found that a single self attention layer trained with gradient descent broke the problem down into hard retrieval and soft composition, which enabled in-context learning and strong overall performance.