INTRODUCING ATTENTION RESIDUALS: RETHINKING DEPTH-WISE AGGREGATION

Summary from the TLDR AI newsletter

Residual connections have long relied on fixed uniform accumulation. Attention Residuals replace standard depth-wise recurrence with learned, input-dependent attention over preceding layers. It enables networks to selectively retrieve past representations, naturally mitigating dilution and hidden-state growth. Attention Residuals have been validated on the Kimi Linear architecture, demonstrating consistent downstream performance gains.