KEEPING LARGE MOE TRAINING WITHIN FIXED GPU MEMORY

Summary from the TLDR AI newsletter

A set of scheduling techniques bounds four major memory bottlenecks in large-scale MoE training, expert dispatch, vocabulary projection, checkpointing, and optimizer state, without approximating the computation.