FLASHATTENTION ON A NAPKIN

Summary from the TLDR AI newsletter

FlashAttention is an extremely performant software implementation of attention. It is hardware aware and prevents unnecessary I/O. However, it can be challenging to understand. This paper aims to clarify and simplify the algorithm with diagrams and explanations.