Daily Edition · Page 4

THE DAILY INDEX

Technology, AI, software, security, crypto and business

ARTICLES & TUTORIALS

TLDR Dev

FROM SSH TO REST: A SECURITY-DRIVEN MODERNIZATION OF SLACK'S EMR DATA PIPELINES

Slack modernized its data infrastructure by migrating over 700 data pipeline jobs from a legacy SSH-based submission model to a REST-based architecture. The previous reliance on direct SSH access to AWS EMR clusters created security vulnerabilities, operational inefficiencies like "zombie" processes, and technical debt that blocked infrastructure upgrades. By using a custom gateway called Quarry and the YARN Distributed Shell feature, the engineering team transitioned workloads into managed containers with server-side lifecycle management.

TLDR IndexPage 4