SCALING CONTEXT REQUIRES RETHINKING ATTENTION

Summary from the TLDR AI newsletter

A new implementation of attention (“Power” attention) allows independent control of state size through a hyperparameter p, solving the challenge of balancing computational costs for long-context training. This mechanism outperforms standard attention on long sequences and allows custom GPU kernels that are 8.6x faster than Flash Attention at 64k context.