Netflix engineers discovered that migrating from Docker to containerd with User Namespace isolation caused severe container startup slowdowns at scale. Each container required idmap mounts for every image layer, generating O(n) kernel system calls and massive global lock contention. The problem was amplified on older Intel Xeon r5.metal AWS instances using a mesh CPU cache architecture, which struggled with cache-line contention compared to newer distributed-cache AMD EPYC or 7th-gen Intel instances. The fix, contributed upstream to containerd, leverages Linux kernel 6.3's recursive bind mount (rbind) support to reduce mount operations from O(n) to O(1) per container. This change was merged into containerd v2.2.0. Netflix also rerouted workloads away from the problematic r5.metal instances as a hardware-level mitigation.

5m read timeFrom cloudnativenow.com
Post cover image
Table of contents
Long Boot TimesToo Many UIDsMulti-Core EngineeringThe CPU BottleneckThe Fix Goes UpstreamRelated

Sort: