PUSHING MIXTURE OF EXPERTS TO THE LIMIT WITH PARAMETER EFFICIENCY

Summary from the TLDR AI newsletter

Mixture of experts (MoEs) is a cool way to increase the capacity of a model without increasing per token runtime. However, they're still tricky to run quickly and fine-tune. Researchers find that if you modify dense model parameters efficiently fine tuning to work with MoEs you can dramatically reduce tuning costs without hurting performance too much.