Kafka is used at Slack as a pub-sub system, playing an essential role in the all-important Job Queue. Slack manages about 0.7 petabytes of data across 10 Kafka clusters, running across hundreds of nodes. We process millions of messages per second with an aggregate throughput of 6.5 Gbps at peak.
Table of contents
Slack’s Kafka journeyWhat is a self-driving Kafka cluster?Kafka 2 projectSplitting up the main Kafka clusterConclusionSort: