Zero-Downtime PyTorch Upgrade in Production: Approaches, Pitfalls and Lessons Chi Zhang | Staff Software Engineer, ML Platform Chen Yang | Sr. Staff Machine Learning Engineer, Applied Science; Lida …

Pint (Python In The News) curates the latest news, updates, and trends in the Python programming language ecosystem. Developers can stay informed about new Python releases, library updates, and community events, as well as discover Python-related articles and resources from around the web. Additionally, Pint highlights Python use cases in various domains such as web development, data science, and machine learning, offering insights into Python's versatility and widespread adoption.

Pinterest Engineering

Pinterest's ML platform team shares a detailed account of upgrading PyTorch from 2.1 to 2.6 in production with zero downtime. The post covers the full cross-stack effort: migrating GPU hosts from Ubuntu 20 to Ubuntu 24 DLAMI with CUDA 12.6 and Nvidia driver 570, bridging breaking LibTorch C++ API changes using a compile-time version macro, resolving TorchScript deadlocks and hangs by disabling JIT profiling mode and NVFuser, handling Caffe2 deprecation with a pinned legacy Docker image, and executing a phased multi-surface rollout. Post-upgrade production issues included DCGM metric loss caused by nv-hostengine resource contention and intermittent CUDA failures traced to a systemd cgroup driver conflict in the Nvidia Container Toolkit, both resolved with targeted configuration fixes.

Zero-Downtime PyTorch Upgrade in Production: Approaches, Pitfalls and Lessons

Get Pinterest Engineering’s stories in your inbox

Bridging Breaking APIs and Deprecated Caffe2