AUXILIARY-LOSS-FREE LOAD BALANCING FOR MOES

Summary from the TLDR AI newsletter

Routing tokens to experts in MoEs helps sparsify the computation. However, learning that routing can be challenging. There is typically a complicated loss structure. This paper shows a novel way to overcome this problem and dramatically improve expert balancing and training stability.