Hugging Face introduces co-located vLLM in TRL to solve GPU inefficiency in GRPO training. Previously, training and inference ran on separate GPUs, causing idle time and resource waste. The new approach runs both processes on the same GPUs, achieving up to 1.73× speedup while maintaining model quality. The solution includes
•14m read time• From huggingface.co
Table of contents
🚀 Introduction 🧨 The Problem 💡 The Opportunity What It Enables 🧩 Design: From Separate Servers to Shared GPUs Server TRL Setup (Top Row) Co-located TRL Setup (Bottom Row) 🛠️ Implementation Notes 📊 Showcase: Co-located vs. Plain TRL Performance Experiment 1: 1.5B Model — Varying Batch Sizes Experiment 2: 1.5B Model — Varying Tensor Parallelism (TP) Experiment 3: 7B Model — Varying Batch Sizes Experiment 4: 7B Model — Varying Tensor Parallelism (TP) 📊 Scaling to 72B Model Sleep Mode in vLLM DeepSpeed Optimizations Accelerate Integration Experiment 5: Qwen2.5-Math-72B — Throughput, Accuracy, and Benchmark Results 🎓 Challenges & Lessons Learned & next steps Challenges Lessons Learned ✅ Conclusion ✅ Give It a Try! 📄 train_grpo_colocate.py 🚀 Introduction🧨 The Problem💡 The Opportunity🧩 Design: From Separate Servers to Shared GPUs🛠️ Implementation Notes📊 Showcase: Co-located vs. Plain TRL Performance📊 Scaling to 72B Model🎓 Challenges & Lessons Learned & next steps✅ Conclusion✅ Give It a Try!Sort: