Best of Distributed Systems — August 2024

  1. 1
    Article
    Avatar of gcgitconnected·2y

    Message Queues in System Design

    Message queues are durable components that support asynchronous communication, helping to decouple events and handle tasks without immediate processing. This allows better scalability and durability, especially under high traffic. Different types of queues like FIFO and priority queues, as well as different models like push-based and pull-based queues, provide versatile solutions for various needs. Examples of message queues include RabbitMQ for versatility, Kafka for high throughput, and Amazon SQS for managed cloud-based services.

  2. 2
    Article
    Avatar of javarevisitedJavarevisited·2y

    How to Design Twitter (X) in a System Design Interview?

    Designing a system like Twitter (X) in a system design interview involves outlining core functionalities such as composing and sharing tweets, following users, and favoriting tweets. Non-functional requirements like scalability, high availability, and stability are crucial for handling large-scale operations. Key aspects include capacity estimation, API design, database design, and understanding queries per second (QPS). Employing a structured approach and utilizing tools like Redis for caching, MySQL for data consistency, and Amazon S3 for media storage are essential. Detailed component design includes load balancers, CDNs, and handling failure scenarios to ensure robust system performance.

  3. 3
    Article
    Avatar of bytebytegoByteByteGo·2y

    EP126: The Ultimate Kafka 101 You Cannot Miss

    This edition of the ByteByteGo newsletter covers several key topics, including a guide to understanding Apache Kafka, tips for efficient API design, an overview of AWS Services, and an advertisement for QA Wolf, an automated testing solution. Kafka is detailed with its core concepts like messages, topics, partitions, producers, consumers, clusters, and use cases. The AWS Services cheat sheet simplifies the exploration of AWS's expansive offerings. Additionally, the newsletter includes 8 practical tips for better API design.

  4. 4
    Article
    Avatar of medium_jsMedium·2y

    How Did LinkedIn Handle 7 Trillion Messages Daily With Apache Kafka?

    LinkedIn uses Apache Kafka to manage and process up to 7 trillion messages daily. They achieve reliability and scalability through a multi-tiered Kafka deployment across multiple data centers, leveraging local and aggregate clusters. LinkedIn ensures message completeness with an internal auditing tool that tracks sent and consumed messages. They maintain a close relationship with the open-source Kafka community by regularly integrating features and patches from their internal branches into the upstream Kafka branch.

  5. 5
    Article
    Avatar of itnextITNEXT·2y

    Orchestration

    Orchestration in software architecture involves a coordinating layer that manages inter-service communication for system-wide use cases. While it enhances debug capabilities and error handling, it can introduce communication overhead and dependency complexities. The orchestrator acts as a facade or mediator, coordinating service interactions and ensuring system consistency. It can simplify the addition of new use cases but complicate service restructuring. Different approaches like SPI and Hierarchy can mitigate some of the challenges.

  6. 6
    Article
    Avatar of communityCommunity Picks·2y

    gRPC From Scratch: Part 1 - Client

    An in-depth exploration of the gRPC protocol, explaining its structure and underlying mechanisms. The post covers the basics of creating a gRPC client in Go, detailing how requests and responses are encoded and decoded at a byte level. It also briefly touches on the topic of gRPC streaming and provides a simple example to get started.

  7. 7
    Article
    Avatar of communityCommunity Picks·2y

    How Netflix Uses Chaos Engineering to Create Resilience Systems 🐒

    Netflix transformed from a DVD rental service to a streaming giant using chaos engineering to build resilient distributed systems. By proactively finding and fixing potential failures through controlled experiments and automation, such as using tools like Chaos Monkey, Netflix ensures minimal downtime and high system availability. Key principles include running tests in production, automating fixes, and controlling the test's blast radius to prevent user impact.

  8. 8
    Article
    Avatar of javarevisitedJavarevisited·2y

    The Art of Asynchronous Communication: Mastering Messaging Queues for Scalable Systems

    Messaging queues are vital for modern applications, enabling asynchronous communication and scalable data processing. Key components include queues, producers, consumers, brokers, and channels. Different types of queues like Point-to-Point, Publish-Subscribe, and Priority Queues serve varied needs. Popular systems include Apache Kafka, RabbitMQ, ApacheMQ, and Amazon SQS. These systems support reliable, scalable, and fault-tolerant communication across distributed systems, crucial for microservices and event-driven architectures.

  9. 9
    Video
    Avatar of communityCommunity Picks·2y

    Using Distributed Locking To Solve Race Conditions (and why a simple solution works better)

    Milan discusses ensuring uniqueness in concurrent environments by tackling race conditions with distributed locking. He illustrates the problem using a user registration example and then explores various solutions, including in-memory locks, semaphores, and database advisory locks. The most effective and straightforward solution presented is using a unique constraint or index in the relational database, which ensures uniqueness without significantly reducing concurrency.

  10. 10
    Article
    Avatar of faunFaun·2y

    | My Tech Radar |11 | Building Reliable Microservices made easy | Lets meet New champion `Dapr.io `

    Dapr (Distributed Application Runtime) is a versatile and powerful open-source project for building reliable and scalable microservices. It offers unified APIs and patterns that simplify communication, state management, and workflow processes across different languages and frameworks. Originally incubated at Microsoft and now part of the Cloud Native Computing Foundation, Dapr integrates industry best practices for security, resiliency, and observability, freeing developers to focus on their core code. This post guides you through setting up a basic Dapr Proof of Concept (POC), from installation to running a simple Go application.

  11. 11
    Article
    Avatar of hnHacker News·2y

    Building a highly-available web service without a database

    Building a web service without a traditional database is now feasible due to advances in storage technology and the introduction of the Raft Consensus algorithm. By leveraging fast, robust disk storage, and cheap RAM, data can be kept entirely in-memory, reducing complexity and improving performance. The architecture involves periodic snapshots and transaction logs to ensure data recovery. For high availability, Raft is used to replicate the service across multiple machines, enabling seamless failover and rolling updates. The example of Screenshotbot shows how this architecture can streamline development and improve robustness.

  12. 12
    Article
    Avatar of microservicesioMicroservices.io·2y

    Evolving an architecture: a problem solving-driven approach

    Explore how to evolve a software architecture using a problem-solving driven approach. Learn from real-world scenarios, such as handling more services than teams after layoffs, and discover frameworks that enable effective architectural decisions.

See all Distributed Systems archives