MIXTURE-OF-MAMBA

Summary from the TLDR AI newsletter

Mixture-of-Mamba introduces modality-aware sparsity to state-space models (SSMs), enabling efficient multi-modal pretraining. It achieves similar performance to Transformers with significantly lower computational cost across text, image, and speech modalities.