HOW NETFLIX KEEPS LARGE GPU JOBS RUNNING

Summary from the TLDR AI newsletter

Netflix described the infrastructure it uses to detect faulty GPUs and network hardware, automatically remediate unhealthy machines, and keep distributed training jobs alive through hardware failures.