Best of Data Streaming — 2025

1. 1
   Article

   Confluent Blog·1y

   The Future of AI Agents is Event-Driven

   AI agents are poised to transform enterprise operations by adopting event-driven architecture. This architectural approach addresses interoperability challenges and enhances scalability. EDA allows agents to operate independently, integrate seamlessly, and adapt workflows dynamically, overcoming the limitations of fixed workflows and tightly coupled systems. It ensures agents can effectively handle complex, interconnected tasks, thereby unlocking their full potential. The article highlights the importance of EDA in creating resilient, scalable AI systems and warns against the risks of outdated architecture in the evolving AI landscape.

   89

   1
2. 2
   Article

   Confluent Blog·35w

   Why Microservices Need Event-Driven Architectures for Agility and Scale

   Event-driven architectures solve critical problems in microservices by replacing synchronous API calls with asynchronous event communication. This approach eliminates bottlenecks, prevents cascading failures, and enables real-time responsiveness. Apache Kafka serves as the central platform for event streaming, allowing services to publish and subscribe to events independently. The shift from REST-based to event-driven microservices delivers faster innovation cycles, improved system resilience, and better customer experiences across industries like finance, e-commerce, and telecommunications.

   65
3. 3
   Article

   The New Stack·38w

   Apache Kafka 4.1: The 3 Big Things Developers Need To Know

   Apache Kafka 4.1 introduces three major developer-focused features: Queues for Kafka (KIP-932) enabling cooperative message consumption with per-message acknowledgment, native JWT-Bearer authentication support eliminating static credentials, and a new Kafka Streams rebalance protocol for better coordination. The release also includes improvements to consumer group protocols, transaction handling, and unified metrics.

   32

   1
4. 4
   Article

   The New Stack·1y

   The New Look and Feel of Apache Kafka 4.0

   Apache Kafka 4.0 introduces significant upgrades, including the replacement of ZooKeeper with KRaft for metadata management, enhancing stability and reducing complexity. The release features Queues for Kafka to allow scaling consumers beyond topic partitions, improved consumer group rebalancing, and new capabilities for code injection and observability. These updates aim to streamline Kafka's operations and improve the developer experience.

   32
5. 5
   Article

   Confluent Blog·1y

   Building Real-Time Multi-Agent AI With Confluent

   Agent Taskflow is utilizing the Confluent data streaming platform to build real-time, multi-agent AI orchestration systems. These systems enable coordinated efforts between AI agents to execute tasks autonomously and in real-time. Key features include a drag-and-drop builder, observability, and fault tolerance. The platform addresses technical challenges in communication, scalability, and governance, ensuring efficient and secure deployment of multi-agent systems.

   30

   1
6. 6
   Article

   SingleStore·1y

   Can a Database Be Faster Than a Formula 1 Engine?

   Formula 1 cars generate vast amounts of real-time telemetry data, which is crucial for strategic decision-making. Each car is fitted with 300 sensors producing 1.1 million data points per second. Teams use this data for simulations, performance analysis, and strategy adjustments. SingleStore's real-time analytics capabilities are highlighted, showcasing its ability to handle high-throughput data streams and provide millisecond response times. The post includes a practical guide for setting up a data ingestion and visualization simulation using SingleStore, Confluent Kafka, and Grafana.

   25

   1
7. 7
   Article

   Data Engineer Things·1y

   Bufstream: Stream Kafka Messages to Iceberg Tables in Minutes

   Bufstream offers a cost-effective and cloud-native alternative to Kafka by using object storage, significantly reducing infrastructure costs. It enhances data quality management by integrating schema validation directly into the broker and seamlessly transforms Kafka messages into Iceberg tables, simplifying the data pipeline. Bufstream also addresses challenges with Kafka's cloud inefficiencies and provides built-in support for schema enforcement and granular access control.

   25
8. 8
   Article

   Community Picks·1y

   ag2ai/faststream: FastStream is a powerful and easy-to-use Python framework for building asynchronous services interacting with event streams such as Apache Kafka, RabbitMQ, NATS and Redis.

   FastStream is a Python framework designed for building asynchronous services that interact with event streams such as Apache Kafka, RabbitMQ, NATS, and Redis. It simplifies the creation of producers and consumers for message queues, leveraging Pydantic for data validation and providing automatic documentation generation. Key features include support for multiple message brokers, intuitive development with full-typed editor support, efficient dependency management, and in-memory testing. FastStream integrates easily with HTTP frameworks like FastAPI, making it suitable for developers at various experience levels.

   24

   1
9. 9
   Article

   The New Stack·1y

   A2A, MCP, Kafka and Flink: The New Stack for AI Agents

   The post discusses the need for a new infrastructure stack to enable AI agents to collaborate effectively. This stack includes four open components: Google’s Agent2Agent (A2A) protocol for agent communication, Anthropic’s Model Context Protocol (MCP) for tool access, Apache Kafka for event-driven communication, and Apache Flink for real-time data processing. By integrating these technologies, AI agents can operate beyond isolated silos, scaling to complex ecosystems that facilitate collaboration, observability, and resilience.

   23

   1
10. 10
    Article

    Dev Genius·1y

    Change Data Capture Tools

    Change Data Capture (CDC) tools automatically track and replicate database changes in real time. Different mechanisms like log-based, trigger-based, query-based, timestamp-based, and hybrid CDC tools are used. Debezium is a popular open-source CDC platform with high scalability, integrating well with Kafka. Other tools like DBConvert Streams, Maxwell Daemon, and Sequin offer various features and integrations for efficient data replication. Challenges such as setup complexity and performance overhead are common with these tools.

    22
11. 11
    Article

    ByteByteGo·1y

    How Canva Collects 25 Billion Events a Day

    Canva collects 25 billion events daily by implementing a robust analytics infrastructure. Key elements include strict schema governance using Protobuf, a unified TypeScript client across platforms, and an AWS Kinesis-backed pipeline for event enrichment and distribution. The system's design focuses on decoupling ingestion from delivery and optimizing costs through compression and clever use of AWS services, maintaining reliability while reducing expenses significantly.

    12
12. 12
    Article

    Confluent Blog·50w

    Build an AI Personalization Engine with Confluent & Databricks

    Confluent and Databricks can be combined to build real-time AI applications by bridging operational and analytical data systems. The tutorial demonstrates creating an AI-powered marketing personalization engine using Tableflow to convert Kafka topics into Delta Lake tables, Apache Flink for stream processing, and Oracle CDC connectors for real-time data ingestion. The example implementation helps a fictional hotel brand identify low-booking properties and generate targeted promotional campaigns using AI-generated content.

    10

    2
13. 13
    Article

    InfoQ·1y

    Kafka 4.0: KRaft Simplifies Architecture

    Apache Kafka 4.0 introduces KRaft mode, eliminating the need for ZooKeeper, thereby simplifying architecture and enhancing scalability. The release also includes a next-generation consumer group protocol for improved performance and early access to point-to-point messaging. Additionally, it updates minimum Java requirements and removes deprecated APIs.

    10