A deep architectural analysis of Apache Kafka's evolution toward cloud-native design, covering tiered storage (KIP-405) with FinOps cost attribution (KIP-1267), the next-generation consumer rebalance protocol (KIP-848) enabling safe Kubernetes autoscaling, Share Groups (KIP-932) for partition-independent parallelism, virtual clusters (KIP-1134) for multi-tenancy, and the emerging diskless storage proposals (KIP-1150, KIP-1163). Includes concrete cost calculations comparing EBS vs S3 storage, PromQL examples for chargeback pipelines, workload decision matrices, and honest assessments of which KIPs are production-ready versus still under community discussion.
Table of contents
Introduction: The Cloud-Native Transition and the "Economic OS"Decoupling Compute and Capacity: The Realities of Tiered StorageClosing the Visibility Gap: FinOps and Cost AttributionGoverning Compute: Elasticity and the Next-Generation ConsumerMulti-Tenancy at Scale: Virtual Clusters vs. Traditional IsolationRedefining Scalability: Share Groups and Queue SemanticsThe Future: The "Diskless" Fork in the RoadConclusionAbout the AuthorSort: