IMPROVING SCALABILITY OF PIPELINE PARALLELISM WITH MEMORY OPTIMIZATION

Summary from the TLDR AI newsletter

Pipeline Parallelism is a great strategy when sharding models across many GPUs. However, it uses substantial memory. This paper, which comes with code, uses offloading to effectively reduce the memory required when the number of shards increases making PP a strong alternative to TP for large model training.