FROM SSH TO REST: A SECURITY-DRIVEN MODERNIZATION OF SLACK'S EMR DATA PIPELINES

Summary from the TLDR Dev newsletter

Slack modernized its data infrastructure by migrating over 700 data pipeline jobs from a legacy SSH-based submission model to a REST-based architecture. The previous reliance on direct SSH access to AWS EMR clusters created security vulnerabilities, operational inefficiencies like "zombie" processes, and technical debt that blocked infrastructure upgrades. By using a custom gateway called Quarry and the YARN Distributed Shell feature, the engineering team transitioned workloads into managed containers with server-side lifecycle management.