FLASH DECODING FOR 8X INFERENCE SPEED UP

Summary from the TLDR AI newsletter

Flash Attention dramatically speeds up the training of large language models. However, when running inference, the batch size is usually 1. This means Flash Attention provides little benefit. Flash Decoding was developed by the same group and brings optimizations to token generation.