ROPE TO NOPE: HYBRID ATTENTION FOR LONG CONTEXT

Summary from the TLDR AI newsletter

The key innovation that enabled Llama 4 to reach 10m+ tokens in context is the alternation between no positional embeddings and rotational positional embeddings. While there are only have benchmarks on Needle in the Haystack, it seems to be a strong confirmation of performance of alternating layers.