TRAINING 1M CONTEXT MODELS WITH NATIVE PYTORCH

Summary from the TLDR AI newsletter

The TorchTitan project has added pass-KV Ring Attention and integrated it with its FSDP-2 training system. Its researchers were able to train Llama 3 8B to use 1M tokens of context with this system on 32 H100s. It is compatible with torch compile, which shows a 10% improvement in tokens per second.