Best of Kafka — August 2024

  1. 1
    Article
    Avatar of gcgitconnected·2y

    Message Queues in System Design

    Message queues are durable components that support asynchronous communication, helping to decouple events and handle tasks without immediate processing. This allows better scalability and durability, especially under high traffic. Different types of queues like FIFO and priority queues, as well as different models like push-based and pull-based queues, provide versatile solutions for various needs. Examples of message queues include RabbitMQ for versatility, Kafka for high throughput, and Amazon SQS for managed cloud-based services.

  2. 2
    Article
    Avatar of bytebytegoByteByteGo·2y

    EP126: The Ultimate Kafka 101 You Cannot Miss

    This edition of the ByteByteGo newsletter covers several key topics, including a guide to understanding Apache Kafka, tips for efficient API design, an overview of AWS Services, and an advertisement for QA Wolf, an automated testing solution. Kafka is detailed with its core concepts like messages, topics, partitions, producers, consumers, clusters, and use cases. The AWS Services cheat sheet simplifies the exploration of AWS's expansive offerings. Additionally, the newsletter includes 8 practical tips for better API design.

  3. 3
    Article
    Avatar of medium_jsMedium·2y

    How Did LinkedIn Handle 7 Trillion Messages Daily With Apache Kafka?

    LinkedIn uses Apache Kafka to manage and process up to 7 trillion messages daily. They achieve reliability and scalability through a multi-tiered Kafka deployment across multiple data centers, leveraging local and aggregate clusters. LinkedIn ensures message completeness with an internal auditing tool that tracks sent and consumed messages. They maintain a close relationship with the open-source Kafka community by regularly integrating features and patches from their internal branches into the upstream Kafka branch.

  4. 4
    Article
    Avatar of devtoDEV·2y

    Introducing AutoMQ: a cloud-native replacement of Apache Kafka

    AutoMQ is a cloud-native replacement for Apache Kafka, designed to address the evolving needs of modern data architectures with a focus on efficiency, scalability, and cost-effectiveness. Originating from a team of open-source pioneers, it offers a unique architecture that decouples storage and computation, leveraging cloud storage to provide significant cost savings and operational efficiency. AutoMQ maintains full compatibility with Kafka, supports multi-cloud environments, and aims to integrate stream data into data lakes to enhance data access and break down silos. The growing community and successful funding highlight its potential impact on the stream storage industry.

  5. 5
    Article
    Avatar of communityCommunity Picks·2y

    Apache Kafka — Important Designs. Filesystem, Zero-copy, and Batching

    Apache Kafka leverages the OS filesystem for data storage, using the page cache to improve performance without adding memory overhead from Java objects. Kafka employs sequential access patterns to optimize read/write operations, benefiting from zero-copy optimization by reducing context switches and data transfers between user and kernel spaces. Batching messages enhances network and disk operation efficiency, ensuring high performance.

  6. 6
    Article
    Avatar of itnextITNEXT·2y

    The streaming bridges — A Kafka, RabbitMQ, MQTT, and CoAP example

    This post provides an in-depth overview of various data streaming protocols including Kafka, RabbitMQ, MQTT, and CoAP, detailing their history, implementation, and use cases. The discussion highlights the differences between push and pull mechanisms, particularly in the context of Kafka and RabbitMQ. Detailed scenarios and examples, including IoT applications, are used to illustrate the practicality and utility of these protocols. Additionally, a practical example using Docker, Apache Spark, and various other tools is provided to demonstrate a comprehensive streaming architecture.

See all Kafka archives