DEPLOY SAFETY: REDUCING CUSTOMER IMPACT FROM CHANGE

Summary from the TLDR Dev newsletter

Slack implemented a "Deploy Safety Program" in 2023 to address the fact that 73% of customer-facing incidents were caused by their own code deployments, as the platform became increasingly mission-critical for users. The program focused on automated detection and remediation within 10 minutes. By January 2025, the initiative successfully reduced customer impact hours by 90% from peak levels through a combination of automated rollback systems, improved monitoring, and cultural changes across their engineering organization.