FREETOKEN: EFFICIENT EDGE-NATIVE MOE SERVING

Summary from the TLDR AI newsletter

FreeToken continuously remaps experts, model state, CPU/GPU work, and agent state reuse to the bandwidth and memory actually available on a personal machine. The authors report support for more than 20 MoE models, from 35B models on an 8GB laptop GPU to a 753B GLM model on one workstation GPU.