KV CACHE LOCALITY: THE HIDDEN VARIABLE IN YOUR LLM SERVING COST

Summary from the TLDR AI newsletter

KV cache locality is a multiplier on existing hardware. The same GPUs serving the same model and handling the same traffic can produce measurably different throughput and latency depending on which GPU gets which request. 'Balanced' and 'efficient' are not the same thing when every request carries thousands of tokens that might already be cached somewhere in the cluster. This post discusses the cost of recomputation, how to measure it, and what changes when load balancers understand token locality.