Best of Distributed Systems — May 2025

1. 1
   Article

   Materialized View·51w

   Kafka: The End of the Beginning

   Apache Kafka has dominated streaming data for over a decade, but innovation has stagnated while batch processing has evolved rapidly. The streaming ecosystem faces challenges with slow growth, long sales cycles, and lack of new ideas. While Kafka's protocol has become the de facto standard, its architecture shows limitations for modern cloud-native requirements. New solutions like S2 are emerging with fresh approaches, and the next decade could see a transition similar to how batch processing moved beyond Hadoop, potentially ushering in a truly cloud-native streaming era.

   293

   6
2. 2
   Article

   ByteByteGo·1y

   How Slack Supports Billions of Daily Messages

   Slack supports billions of daily messages by maintaining over five million simultaneous WebSocket sessions at peak hours, addressing the architectural challenge of real-time interaction. Initially developed from a video game infrastructure, Slack's architecture separates message propagation from business logic and adopted a push-first model using WebSockets. Significant challenges included scaling session initialization for large teams and ensuring message persistence and order through atomic broadcast approximations. Innovations like Flannel and architectural shifts improved the system's resilience and efficiency at scale, embracing complexity where necessary while simplifying other areas.

   47

   1
3. 3
   Article

   Platformatic·1y

   Scale WebSockets in Kubernetes with Node.js

   The post discusses the challenges and solutions involved in scaling WebSocket services in Kubernetes, focusing on connection management, distributed state, and message delivery guarantees. It offers best practices such as stateless server usage with consistent hashing, cross-node communication, and kernel tuning, as well as compares WebSockets with other real-time protocols like SSE and MQTT.

   38
4. 4
   Article

   Awesome Go·1y

   Build your own ResponseWriter: safer HTTP in Go

   Learn how to build a custom ResponseWriter in Go to prevent common HTTP handling errors, such as forgetting status codes or improperly modifying headers. This guide shows how to wrap the ResponseWriter to enforce custom rules, making HTTP handlers more secure and maintainable.

   35
5. 5
   Article

   Baeldung·1y

   Flexible Pub/Sub Messaging With Spring Boot and Dapr

   Learn how to integrate Dapr with Spring Boot to create a flexible publish/subscribe messaging system. This approach decouples application code from specific message brokers, allowing seamless swapping without modifying code. The article provides a tutorial on setting up and testing a ride-hailing application, showcasing Dapr's capabilities for distributed messaging and the benefits of running tests locally using Testcontainers.

   11