JOINTLY TRAINING LARGE MULTIMODAL MODELS

Summary from the TLDR AI newsletter

Models are usually trained separately on specific tasks (e.g., language vs image generation). The newly proposed joint autoregressive mixture (JAM) algorithm combines disparate models with a clever interleaved cross attention and gentle fine-tuning achieves dominant performance in many multimodal tasks.