HOW SLACK PROCESSES 33,000 JOBS PER SECOND

Summary from the TLDR Web Dev newsletter

Slack manages many tasks asynchronously, including push notifications and billing. It previously used Redis for task queuing, but as Slack scaled, it faced limitations such as memory constraints and coupling between workers and Redis instances. To address these issues, Slack integrated Apache Kafka, added Kafkagate as a new scheduler, and used JQRelay to relay jobs from Kafka to Redis. This improved Slack’s scalability and durability and reduced complexity in managing job queues.