Recent developments in reinforcement learning for large language models (LLMs) focus on improving reasoning abilities. While new models like GPT-4.5 and Llama 4 were released, their conventional training methods faced muted responses. Competing models by xAI and Anthropic have advanced reasoning features. OpenAI’s o3 model used extensive compute resources through tailored reinforcement learning for reasoning tasks. The article delves into the GRPO algorithm, the effect of RLHF to align LLMs, and insight from recent research on improving reasoning capabilities in LLMs.
Table of contents
Understanding reasoning modelsRLHF basics: where it all startedA brief introduction to PPO: RL’s workhorse algorithmRL algorithms: from PPO to GRPORL reward modeling: from RLHF to RLVRHow the DeepSeek-R1 reasoning models were trainedLessons from recent RL papers on training reasoning modelsNoteworthy research papers on training reasoning modelsSort: