HOW TO THINK ABOUT GPUS

Summary from the TLDR AI newsletter

This article looks at how Nvidia GPUs work, how they're networked together, and what that means for large language models, especially compared to TPUs. A modern machine learning GPU is basically a bunch of compute cores that specialize in matrix multiplication connected to a stick of fast memory. TPUs are dedicated matrix multiplication machines that are cheap, predictable, and great if used as intended. Networking is one of the areas where GPUs and TPUs differ the most - TPUs are connected in 2D or 3D tori, where each TPU is only connected to its neighbors, while GPUs use a more tree-based switching network.