MAXIMIZE TRAINING THROUGHPUT WITH PYTORCH FSDP

Summary from the TLDR AI newsletter

Teams from Meta and IBM have combined to achieve 57% MFU, using massive A100 and H100 clusters to quickly train powerful models in parallel.