THE MAMBA IN THE LLAMA: DISTILLING AND ACCELERATING HYBRID MODELS

Summary from the TLDR AI newsletter

Mambda models are hard to accelerate with speculative decoding because of the lack of a KV cache for backtracking. This paper, from some of the original authors, showcases some novel acceleration algorithms and distillation methods.