HOW TO THINK ABOUT GPUS

Summary from the TLDR Web Dev newsletter

This article explains the architecture of GPUs, focusing on Streaming Multiprocessors (SMs), CUDA Cores, Tensor Cores, and memory hierarchy, and summarizes the specifications of different GPU generations. It then compares GPUs and TPUs at the chip level and covers GPU networking, detailing intra-node and cross-node collectives and their performance implications.