BAM DENSE TO MOE UPCYCLING

Summary from the TLDR AI newsletter

This method allows dense models' FFN and Attention layers to be upcycled into a Mixture of Experts (MoE) model for further training. This saves a dramatic amount of compute cost while maintaining downstream performance.