HOW TO MAKE LLMS 3X FASTER

Summary from the TLDR newsletter

Speculative decoding is a technique that converts a GPU's unused math units into output. It involves using a second, much smaller model that produces several candidate tokens in advance. The large model evaluates all of them in a single forward pass. This results in up to three times faster generation.