IBM THROWS HAT IN LANGUAGE MODEL ARENA WITH OPEN SOURCE MOE MODELS

Summary from the TLDR AI newsletter

Mixture-of-Experts models are sparse models, which means they save on computation while not sacrificing accuracy. They're tricky to train. IBM has released several along with their architecture and some code.