1 BILLION TOKENS IN CONTEXT

Summary from the TLDR AI newsletter

It seems like dilated attention now finally works! This is an exciting advancement if it holds at scale, because it means that we can essentially treat entire corpi of documents as context. All of the Python code on GitHub is only a few hundred billion tokens. LongNet is a linear attention variant, but they seem to have stabilized training, and it doesn’t sacrifice performance on shorter sequences.