A GUIDE TO LLM INFERENCE AND PERFORMANCE

Summary from the TLDR newsletter

Using the full power of the GPU during LLM inference requires knowing if the inference is compute-bound or memory-bound. Calculating the operations per byte possible on a given GPU and comparing it to the arithmetic of a model's attention layers reveals where the bottleneck is. This information can be used to pick an appropriate GPU for model inference and use techniques like batching to better utilize GPU resources.