Under the hood: Redpanda Cloud Topics architecture

Redpanda's Cloud Topics architecture separates metadata storage (Raft logs) from data storage (object storage like S3 or GCS), enabling cost-efficient streaming without Cross-AZ replication overhead. On the write path, incoming data is batched in memory and flushed as L0 files to object storage, with only lightweight placeholder batches written to Raft logs to preserve transaction and idempotency guarantees. A background Reconciler process continuously reorganizes L0 files into larger, partition-sorted L1 files for efficient historical reads. The read path routes requests based on a Last Reconciled Offset: recent data is served from L0 (or memory cache), while historical data is served from optimized L1 files. Cloud Topics is now generally available in Redpanda Streaming 26.1.