Kafka now supports native queue semantics via the Share Consumer API (KIP-932), now generally available on Confluent Cloud and coming to Confluent Platform 8.2 with Apache Kafka 4.2. Share groups allow multiple consumers to process messages from the same topic regardless of partition count, breaking the traditional 1:1 partition-to-consumer constraint. The broker manages per-message acquisition locks (default 30s), enabling consumers to acknowledge, release, reject, or renew individual messages. This eliminates the need to run separate messaging systems for streaming and task queue workloads. Key use cases include command invocation, job processing, and event-driven workflows requiring elastic scaling. The new KafkaShareConsumer class is designed as a near drop-in replacement for KafkaConsumer. Future roadmap includes DLQ support, exactly-once semantics, exponential back-off, and key-based ordering.

7m read timeFrom confluent.io
Post cover image
Table of contents
The Infrastructure Consolidation ChallengeWhat Is Queues for Kafka?How It WorksQueues for Kafka on Confluent Cloud and Confluent PlatformWhen to Use Share Consumer vs Consumer APIGetting Started – Try Queues for Kafka TodayWhat's Next

Sort: