MIXTURE OF BLOCK ATTENTION FOR LONG CONTEXT LLMS

Summary from the TLDR AI newsletter

Moonshot has a great algorithm that is comparable to NSA in that it enables long context language modeling more effectively.