Learn how speculative decoding in vLLM can significantly increase throughput without altering a model's output quality, resulting in 19% cost savings at scale for enterprise AI. This post benchmarks gpt-oss-120B with Eagle3 speculative decoding on vLLM and demonstrates consistent throughput and latency improvements across varying concurrency levels, datasets, tensor-parallelism settings, and draft-token budgets.

Rhdev is a blog and resource hub dedicated to Ruby on Rails development, a popular web application framework written in Ruby. Developers can explore tutorials, best practices, and case studies for building web applications with Ruby on Rails. Additionally, Rhdev covers topics such as ActiveRecord ORM, RESTful APIs, and frontend integration using JavaScript frameworks, offering insights for both beginners and experienced Rails developers.

Red Hat Developer

Benchmarking results show that Eagle3 speculative decoding in vLLM delivers consistent throughput and latency improvements for the gpt-oss-120B mixture-of-experts model, even at up to 200 concurrent requests — contradicting the common assumption that speculative decoding only helps at low concurrency. Across three real-world datasets (ShareGPT, MLPerf, SWE-bench) and two tensor-parallelism settings, output throughput improves by 10–21% and end-to-end latency drops by 12–20%. On code-heavy SWE-bench workloads, the gains translate to a 19.4% reduction in cost per 1M output tokens on H200 GPUs. The analysis also covers optimal draft token count, finding 2–3 draft tokens to be the sweet spot, with 4 tokens adding overhead without proportional benefit.

Performance improvements with speculative decoding in vLLM for gpt-oss

Why it matters: 19% cost savings at scale

Benchmarking on ShareGPT: Speculative decoding versus baseline

Tuning the draft token count: How many speculative tokens?

Validating on MLPerf: Cross-dataset and cross-TP confirmation

Cost implications for production deployments