SPEED UP SPECULATIVE DECODING WITHOUT A DRAFT MODEL

Summary from the TLDR AI newsletter

Language model inference is generally slow because these models are heavily memory-bound. So people have introduced speculative decoding using a smaller draft model to “run ahead” and offer suggestions to the large model. This works well, but it’s complicated to implement, and finding a good draft model is tough. Instead, you can create a Medusa model that acts as its own draft by splitting off activations from various internal stages. This speeds up Llama 2x without compromising performance.