PAGEDATTENTION: VIRTUAL MEMORY FOR THE KV CACHE

Summary from the TLDR Data newsletter

PagedAttention brings virtual-memory-style paging to LLM KV caches by replacing contiguous per-request allocation with fixed-size blocks and a block table. That design eliminates most cache fragmentation, raises utilization to roughly 96%, supports copy-on-write for shared prompts, and can serve 2–4x more users per GPU without changing attention outputs.