Tracking Down Mysterious ML Training Stalls Chen Yang, Andrew Yu, Shunyao Li, Pong Eksombatchai, Mark Molinaro Introduction Pinterest’s ML training platform, MLEnv, is a collection of external …

Pint (Python In The News) curates the latest news, updates, and trends in the Python programming language ecosystem. Developers can stay informed about new Python releases, library updates, and community events, as well as discover Python-related articles and resources from around the web. Additionally, Pint highlights Python use cases in various domains such as web development, data science, and machine learning, offering insights into Python's versatility and widespread adoption.

Pinterest Engineering

Pinterest engineers debugged a mysterious 50% throughput drop during a PyTorch upgrade that manifested as periodic training stalls. Through systematic investigation using profiling tools, they identified two root causes: a PyTorch dispatch mode interfering with torch.compile, and Ray's monitoring process using psutil.memory_full_info which locked page tables every 2.5 seconds. By disabling the problematic FlopCountMode and removing detailed memory statistics collection, they achieved a 2.4x improvement over the degraded performance, ultimately gaining 20% speed up compared to the original version.

Tracking Down Mysterious ML Training Stalls

Get Pinterest Engineering’s stories in your inbox