Redpanda's Cloud Topics feature introduced a new performance bottleneck by splitting the write path into two stages: metadata replication and payload upload to cloud object storage. Since object storage latency can be 100x higher than local replication, throughput collapsed. Applying Little's Law (Throughput = Latency × Concurrency), the team resolved this by adding an extra queuing layer before the replication stage, enabling parallel uploads from a single producer. This preserved idempotent ordering and durability guarantees while unlocking GB/s-scale throughput without requiring producer configuration changes.

6m read timeFrom redpanda.com
Post cover image
Table of contents
How Redpanda handles low-latency produce processing #How Cloud Topics handles produce processing #Solving the bottleneck #Learn more about Cloud Topics #

Sort: