WHY IS SPECULATIVE DECODING FAST?

Summary from the TLDR AI newsletter

Speculative Decoding is a technique where a small draft model proposes a likely draft sequence, and a big model verifies that sequence in parallel in a single forward pass. It enables the ability to look at many tokens at once and accept the ones that would have been generated by the big model. When the batch size is smaller than the optimal number of tokens, speculative decoding allows models to use 'free' compute to complete sequences faster. However, at high throughput, you could actually be wasting compute when you could be using that time to move memory.