HOW LLM INFERENCE WORKS

Summary from the TLDR AI newsletter

This post walks through the inference pipeline from tokenization and embeddings through stacked self-attention layers, then splits generation into two distinct phases on the same GPU: compute-bound prefill that processes all input tokens in parallel and memory-bound decode that emits one token at a time.