HOW GOOGLE MADE THEIR GEMMA LLM 3X FASTER

Summary from the TLDR Dev newsletter

Google's Gemma model achieved a three-fold increase in inference speed using speculative decoding. This technique involves a tiny, fast 'Drafter' predicting tokens that the main model verifies in parallel to efficiently use the GPU and save memory.