MULTIMODAL PATHWAY: TRANSFORMERS WITH IRRELEVANT DATA FROM OTHER MODALITIES

Summary from the TLDR AI newsletter

This project proposes a novel approach for enhancing transformers that uses unrelated data from different modalities, such as improving an image model using audio data. Multimodal Pathway uniquely connects transformers of two different modalities, allowing a target modality to benefit from the strengths of another.