HOW WE BUILT A SELF-HEALING SYSTEM TO SURVIVE A TERRIFYING CONCURRENCY BUG AT NETFLIX

Summary from the TLDR Web Dev newsletter

A concurrency bug in a Netflix internal library was causing CPUs to die at a rate of one every two minutes. Due to the bug being unfixable until Monday, Netflix engineers implemented a self-healing system by pinning their cluster size to the maximum and randomly terminating a few instances every 15 minutes. This let the system stay running until Monday when it could be fixed.