SPARSE BY DESIGN

Summary from the TLDR AI newsletter

Kimi K3 activates 16 of 896 experts per token. While the total parameters have grown, the active parameters have barely grown at all over the past three releases. The strategy appears to be to keep per-token compute roughly flat while relentlessly inflating total capacity. At a fixed training compute budget, more experts means lower loss, and the model learns more from the same FLOPs. The open source labs have discovered that the cheapest way to buy intelligence is to spend capacity.