HOW TO TRAIN A MODEL ON 10K H100 GPUS?

Summary from the TLDR newsletter

This post summarizes some of the common knowledge among people who train large-scale models. It covers parallelizing and memory-saving tricks, how to communicate state between these GPUs as quickly as possible, and how to recover from failures as quickly as possible. A list of resources for further reading is available.