RECENT DEVELOPMENTS IN LLM ARCHITECTURES: KV SHARING, MHC, AND COMPRESSED ATTENTION

Summary from the TLDR AI newsletter

KV-cache size, memory traffic, and attention cost quickly become the main constraints as reasoning models and agent workflows keep more tokens around for longer. LLM developers are adding a growing number of architecture tricks to reduce costs. Most of the changes look like small tweaks, but some are quite intricate design changes. This article looks at these architecture changes with a focus on what changes inside the transformer block, residual stream, KV cache, and attention computation.