AI Researcher's New Trick: Train LLMs To Explore On "Hard" Tokens

Researchers have developed new methods to improve reinforcement learning with verifiable rewards (RLVR) for large language models by focusing training on high-entropy "forking" tokens where models make critical decisions. Two approaches are explored: completely ignoring the 80% lowest-entropy tokens during training to reduce computational cost while improving accuracy, and adding bonus rewards to high-entropy tokens to encourage exploration and prevent models from collapsing to single solutions. Both methods show significant improvements in mathematical reasoning tasks by concentrating learning signals on pivotal decision points rather than spreading them across all tokens uniformly.