The Enterprise Ready Conference for engineering leaders (Sponsored)

ByteByteGo provides tutorials, articles, and resources for learning and mastering the Go programming language, covering topics such as syntax, concurrency, and best practices. Developers can learn about Go programming fundamentals, web development with Go, and building scalable applications using Go's powerful features and standard library.

ByteByteGo

Walmart's Apache Kafka setup processes trillions of messages daily with a 99.99% availability rate, supporting critical data movement, event-driven microservices, and streaming analytics. The team addressed challenges like consumer rebalancing, poison pill messages, and costs by designing a Message Proxy Service (MPS). This service decouples Kafka message consumption from its partition-based model, allowing consumer applications to scale independently and handling consumer failures effectively.

The Trillion Message Kafka Setup at Walmart

Challenges with Kafka at Walmart’s Scale

Designing the Messaging Proxy Service (MPS)

This is just mind blowing that they have all this microservices all working and synchronizing together and also able to deliver this massive amount of messages

So many poison pills I’ve hit with Kafka and messages serialized with Avro, multiple versions, and only max 1 million msg/day.
LE: punctuation fix