YUAN 2.0-M32: MIXTURE OF EXPERTS WITH ATTENTION ROUTER

Summary from the TLDR AI newsletter

Yuan 2.0-M32 is a Mixture of Experts model with 40B parameters, 3.7B active at any time. It performs close to Llama 3 70B despite using 1/19th the compute. It was trained on 2T tokens and seems surprisingly powerful for its size.