Qwen Team Finds Training Only 20% High-Entropy Tokens Beats Full-Gradient RLVR for LLM Reasoning

Forum topic · 小凯 · 2026-05-11

Summary

A study by the Qwen team (Alibaba) and Tsinghua's LeapLab, titled "Beyond the 80/20 Rule" (arXiv:2506.01939), shows that in RLVR training for LLM reasoning, roughly 80% of token gradients are essentially noise. The researchers find that only about 20% of chain-of-thought tokens—high-entropy "fork" tokens that decide the direction of reasoning—actually change during training, while low-entropy follower tokens remain nearly untouched. Training only on the top 20% highest-entropy tokens matches or exceeds full-gradient RLVR: Qwen3-32B improves by +7.71 on AIME'24 and +11.04 on AIME'25, reaching 68.1 on AIME'24 with extended context—a SOTA result for models under 600B parameters. Conversely, training only on the 80% low-entropy tokens sharply degrades performance, and 20% is the optimal ratio across tested scales (8B/14B/32B), with benefits growing with model size. The findings suggest RL training compute could be cut by up to 80% by masking gradients on low-entropy tokens.

The Qwen team (Alibaba) and Tsinghua University's LeapLab published a counterintuitive finding about reinforcement learning with verifiable rewards (RLVR): most token gradients in RL training are noise, and reasoning ability hinges on a small minority of high-entropy "fork" tokens.

Key points

~20% of CoT tokens are high-entropy "fork" tokens that determine the direction of reasoning (e.g., whether to try a different approach, verify, or backtrack). The remaining ~80% are low-entropy follower tokens that merely fill in reasoning details (e.g., after "2 + 2 =" the answer "4" is nearly certain).
RLVR mostly only moves high-entropy tokens. Entropy tracking shows low-entropy token distributions are barely altered during training—yet gradients are still computed for all tokens.
Training only the top 20% high-entropy tokens (masking the other 80%) matches or beats full-gradient RLVR:

| Model | 100% tokens | 20% high-entropy tokens | |---|---|---| | Qwen3-8B | baseline | comparable | | Qwen3-14B | baseline | +5.21 (AIME'24), +4.79 (AIME'25) | | Qwen3-32B | baseline | +7.71 (AIME'24), +11.04 (AIME'25) |

With maximum length extended to 29k, the 32B model trained on 20% tokens reaches 68.1 on AIME'24—reported as SOTA among models under 600B parameters.
Training only the 80% low-entropy tokens severely degrades performance, suggesting their gradients may actively introduce noise.
20% is the sweet spot: 10% under-explores key forks; 50%+ dilutes training with uninformative tokens.
Benefits scale with model size: gains grow from 8B (flat) to 14B (+5) to 32B (+11), implying larger models benefit more from targeting key tokens.

Why high-entropy tokens matter

Low-entropy tokens act like "autopilot"; high-entropy tokens are strategic decision points—analogous to the few critical moves that decide a chess game. Focusing RL updates on these forks makes training more efficient and effective.

Implications

Up to ~80% compute savings on gradient computation and backpropagation, enabling more experiments per budget and lower iteration costs.
The result echoes related work: TokenSkip (40% of CoT tokens are compressible), DAST (difficulty-adaptive compute), showing chain-of-thought contains substantial redundancy—in content, compute, and gradients.
The author predicts entropy-based gradient masking will become a standard feature in RL frameworks (veRL, OpenRLHF, trl) by 2026, since implementation is just a few lines of masking code.

Limitations and open questions

Validated mainly on math reasoning (AIME); whether 20% is optimal for code, science, or multimodal tasks is untested.
Could better "forkiness" metrics than standard entropy $H = -\sum p_i \log p_i$ exist?
Potential combinations with MRT's progress rewards and E3's exploration incentives.
A dynamic (rather than fixed) token ratio—higher early for exploration, lower later for precision—might outperform the static 20%.

Paper details

Title: Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning
Authors: Shenzhi Wang, Le Yu, Chang Gao, Chujie Zheng, Shixuan Liu, Rui Lu, Kai Dang, Xionghui Chen, Jianxin Yang, Zhenru Zhang, Yuqiong Liu, An Yang, Andrew Zhao, Yang Yue, Shiji Song, Bowen Yu, Gao Huang, Junyang Lin
Institutions: Qwen Team (Alibaba Inc.), LeapLab (Tsinghua University)
arXiv: 2506.01939 (2025-06-02)
Project page: https://shenzhi-wang.github.io/high-entropy-minority-tokens-rlvr