HOW PAYPAL SCALED KAFKA TO 1.3 TRILLION MESSAGES A DAY

Summary from the TLDR Web Dev newsletter

Apache Kafka is used to move messages between backend components. Paypal’s Kafka fleet consists of over 85 clusters that process over 100 billion messages per day. This article goes over how Paypal introduced Kafka-specific client libraries, config services, and more, in order to scale Kafka to such a large number of clusters. Furthermore, its Kafka clusters are able to handle peaks in usage smoothly, with Black Friday bringing in 1.3 trillion messages a day.