LLM TRAINING PUZZLES

Summary from the TLDR AI newsletter

While we are all waiting for our 22k H100s to be delivered, we should probably practice training large models. Sasha Rush has another set of puzzles, but instead of GPUs or tensor shapes, he walks us through model parallelism, gradient accumulation, and more!