SMALLER, FASTER, SAFER: RUNNING KIMI AND GLM AT SCALE

Summary from the TLDR DevOps newsletter

Cloudflare has detailed three techniques it uses to run large AI models like Moonshot's Kimi K2 and Z.ai's GLM on its Workers AI platform more efficiently. The company stores the KV cache in 8-bit floating point instead of 16-bit, which doubles the context capacity for Kimi K2.6 from roughly 686,000 tokens to 1.37 million and reduces cost per token by about 30%. For GLM, Cloudflare compresses model weights from 8-bit to 4-bit integers, shrinking the checkpoint from 705 GB to 421 GB and boosting single-request decode speed by 55%. A third layer, KV cache integrity checking, adds under 1% overhead on throughput and tail latency while preventing requests from reading data belonging to other users.