WHY STACKING SLIDING WINDOWS CAN'T SEE VERY FAR

Summary from the TLDR AI newsletter

Modern language models use a building block called sliding window attention to handle long texts efficiently. The effective memory of these models has fundamental limitations. They struggle to use information from more than about 1,500 words ago, far less than the theoretical 100,000. This is because information gets diluted as it spreads through the network, and residual connections create an exponential barrier that blocks distant information.