Qwen Team's 80/20 Finding: Training on Only 20% High-Entropy Tokens Beats Full-Gradient RLVR

Forum topic · 小凯 · 2026-05-11

Summary

A study by the Qwen team (Alibaba) and Tsinghua University's LeapLab, 'Beyond the 80/20 Rule' (arXiv:2506.01939), shows that in RLVR training for LLM reasoning, roughly 80% of token gradients are effectively noise. Only about 20% of tokens—high-entropy 'fork' tokens that decide the direction of a reasoning chain—drive meaningful learning. By masking gradients to keep only the top 20% highest-entropy tokens, the authors match or beat standard full-gradient RLVR across Qwen3-8B/14B/32B: the 32B model gains +7.71 on AIME'24 and +11.04 on AIME'25, reaching 68.1 on AIME'24 with extended context—reported as SOTA for models under 600B parameters. Training only on the 80% low-entropy tokens instead sharply degrades performance. The findings imply potential ~80% reductions in RL training compute, benefits that scale with model size, and a simple implementation via entropy-based gradient masking. This forum post summarizes the paper, presents ablations over token ratios, and discusses limitations and future directions.

Key points

A forum post on zhichai.net discusses the paper "Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning" (arXiv:2506.01939) by the Qwen team (Alibaba) and LeapLab (Tsinghua University).

Core claim: In RLVR (reinforcement learning with verifiable rewards), ~80% of tokens are low-entropy "follower" tokens whose gradients are essentially noise. The real drivers of reasoning ability are the ~20% high-entropy "fork" tokens—decision points in a chain-of-thought that determine which reasoning path the model takes (e.g., "let me try another approach...").
Entropy analysis: Tracking token entropy during RLVR shows high-entropy fork tokens shift significantly during training while low-entropy tokens barely change—most compute updates tokens that don't need training.

Experimental results (training on only top 20% high-entropy tokens, masking the rest)

| Model | vs. 100% token RLVR | |---|---| | Qwen3-8B | Comparable on AIME'24 / AIME'25 | | Qwen3-14B | +5.21 AIME'24, +4.79 AIME'25 | | Qwen3-32B | +7.71 AIME'24, +11.04 AIME'25 |

With maximum length extended to ~29k, Qwen3-32B reaches 68.1 on AIME'24, reported as SOTA for models under 600B parameters (56.7 on AIME'25; 63.5 AIME'24 without length extension).
Training only on the 80% low-entropy tokens instead causes severe performance drops—their gradients are not just useless but potentially harmful noise.
Ratio ablation: 10% of tokens underperforms (insufficient exploration); 20% is optimal; 50% degrades (low-entropy tokens dilute exploration); 100% is the baseline.
Scale effect: gains from the 20% strategy grow with model size (0 at 8B, +5 at 14B, +11 at 32B), suggesting larger models benefit most from precisely targeted training.

Implications

Cost: gradient computation and backpropagation could be cut by ~80%, enabling ~5x more experiments per budget and making RL training accessible to smaller teams.
Related work: echoes TokenSkip (40% of CoT tokens are redundant content) and DAST (difficulty-adaptive compute)—a common theme that reasoning chains contain large redundancy, whether in content, computation, or gradients.
Simple implementation: entropy-based gradient masking before backpropagation is only a few lines of code; the author predicts entropy-based token selection will become standard in RL frameworks (veRL, OpenRLHF, trl) by 2026.

Limitations raised in the post

The 20% ratio is validated mainly on math reasoning (AIME); optimality in code, science, or multimodal domains is untested.
Standard entropy $H = -\sum p_i \log p_i$ may not be the best fork-measure; alternatives like conditional entropy change rates could help.
Possible combinations with episode-level progress rewards (MRT) and exploration methods (E3).
A fixed 20% may be suboptimal—a dynamic ratio (more exploration early, more precision later) could improve results.

Paper details

Authors: Shenzhi Wang, Le Yu, Chang Gao, Chujie Zheng, Shixuan Liu, Rui Lu, Kai Dang, Xionghui Chen, Jianxin Yang, Zhenru Zhang, Yuqiong Liu, An Yang, Andrew Zhao, Yang Yue, Shiji Song, Bowen Yu, Gao Huang, Junyang Lin
arXiv: 2506.01939 (2025-06-02)
Project page: https://shenzhi-wang.github.io/high-entropy-minority-tokens-rlvr