A SINGLE DNS RACE CONDITION BROUGHT AMAZON'S CLOUD EMPIRE TO ITS KNEES

Summary from the TLDR newsletter

Amazon has published a detailed postmortem that explains the recent day-long outage that disrupted major websites and services. The root cause was a race condition in DynamoDB's automated DNS management system that left an empty DNS record for the service's regional endpoint. The race condition occurred when one DNS Enactor experienced 'unusually high delays' while the DNS Planner continued generating new plans. AWS has disabled the DynamoDB DNS Planner and DNS Enactor automation worldwide until safeguards are put in place.