PREFILL-AS-A-SERVICE: KVCACHE OF NEXT-GENERATION MODELS COULD GO CROSS-DATACENTER

Summary from the TLDR AI newsletter

Prefill-as-a-Service (PrfaaS) is a cross-datacenter serving architecture that selectively offloads long-context prefill to standalone, compute-dense prefill clusters and transfers the resulting KVCache over commodity Ethernet to local PD clusters for decode. It combines model-side KV efficiency with system-side selective offloading, bandwidth-aware scheduling, and cache-aware request placement. The design removes the requirement that heterogeneous accelerators share the same low-latency RDMA fabric and enables independent scaling of prefill and decode capacity across loosely coupled clusters. A PrfaaS-augmented heterogeneous deployment achieves higher serving throughput while consuming only modest cross-datacenter bandwidth.