TWIST: IMPROVING SPEECH LANGUAGE MODELS WITH TEXTUAL WARM-START

Summary from the TLDR AI newsletter

This work introduces TWIST, a method for training Speech Language Models (SpeechLMs) using a warm-start from pretrained textual models. TWIST outperforms starting from scratch, with empirical analysis emphasizing the significance of model and dataset scale. The study presents the largest SpeechLM to date and introduces spoken versions of a textual benchmark for improved evaluation and future research advancements.