BENCHMARKING MULTIHEADED ATTENTION IMPLEMENTATIONS (JUPYTER NOTEBOOK)

Summary from the TLDR AI newsletter

There is an almost 10x difference in speed between different implementations of the popular multi-headed attention module used in modern LLMs. This notebook shows a few and benchmarks their performance.