FAULT TOLERANT LLAMA: TRAINING WITH 2000 SYNTHETIC FAILURES EVERY ~15 SECONDS AND NO CHECKPOINTS ON CRUSOE L40S

Summary from the TLDR AI newsletter

Researchers used torchft and torchtitan to train a model in a real-world environment with extreme synthetic failure rates to prove the reliability and correctness of fault tolerant training.