DIFFERENTIAL TRANSFORMER

Summary from the TLDR AI newsletter

Transformers tend to over-allocate attention to irrelevant context. This work introduces Diff Transformer, which amplifies attention to the relevant context while canceling noise. Specifically, the differential attention mechanism calculates attention scores as the difference between two separate softmax attention maps. The subtraction cancels noise, promoting the emergence of sparse attention patterns