WE REVERSE-ENGINEERED FLASH ATTENTION 4

Summary from the TLDR AI newsletter

Flash Attention 4 is a brand new, highly optimized CUDA kernel that speeds up the attention calculations in transformers, which are the main mathematical operations that bottleneck AI models like ChatGPT. However, the improvements from FA4 are not from clever new math but from improvements in how an incredibly complex async pipeline splits computations across 32-thread ‘warps'.