STREAMINGLLM

Summary from the TLDR AI newsletter

Language models are limited by their context length. The context length is usually limited by compute hardware and clever algorithmic updates. This is an algorithmic update that streams the tokens through the attention mechanism to allow theoretically infinite context window size. Usually, these claims fall over at scale, but this one seems robust as it can work on existing pre-trained models without fine-tuning. Will it make the forgotten middle issue worse though?