700 TPS ON KIMI K3: A CASE FOR TPU MEGAKERNELS

Summary from the TLDR AI newsletter

inferact/tpu-megakernels is a collection of megakernels for TPU v7. Its Kimi K3 implementation delivers over 700 tokens per second with speculative decoding. Without speculative decoding, its megakernels for K3 and Qwen 3.8 27B deliver roughly 1.4 to 2x the decode throughput of the GB200 baseline at batch sizes 1 through 8. This post explains the design and why megakernels and TPUs are a good fit.