Running Kafka Connect on Kubernetes introduces friction due to competing scheduling systems, lack of connector isolation, and a mutable REST-based control plane. The post details specific pain points: resource contention between connectors sharing a JVM, inability to use Kubernetes health checks per connector, scaling complexity, and security risks from intra-process isolation failures. A vision is proposed for a Kubernetes-native redesign where a Kubernetes operator replaces Connect's internal clustering layer, spinning up one pod per connector task. This enables true workload isolation, Kubernetes-native lifecycle management via custom resources, immutable GitOps-friendly configuration, and opens the door to GraalVM native compilation and multi-language connectors.
Sort: