MIXTURE-OF-TRANSFORMERS: A SPARSE AND SCALABLE ARCHITECTURE FOR MULTI-MODAL FOUNDATION MODELS

Summary from the TLDR AI newsletter

The Mixture-of-Transformers (MoT) architecture introduces a sparse multi-modal transformer that decouples parameters by modality (text, images, and speech), enabling efficient processing while maintaining performance quality. Across multiple evaluations, including the Chameleon 7B and Transfusion settings, MoT achieves comparable or better performance than dense baselines while using significantly fewer computational resources - as low as 37.2% of the FLOPs for speech processing and 47.2% of the wall-clock time for image generation.