PAGEDATTENTION: VIRTUAL MEMORY FOR THE KV CACHE

Summary from the TLDR AI newsletter

The KV cache is a per-request store of attention keys and values that lets models avoid recomputing them at every decoding step. It grows linearly with sequence length, and at long contexts, it eats more GPU memory than the model weights themselves, so it's a scarce resource. KV cache wastes a lot of memory by default. This article talks about how PagedAttention implements the idea of virtual memory in a way that attention kernels can still work with.