Beyond the 80/20 Rule: How High-Entropy Minority Tokens Drive RLVR Reasoning Gains

Forum topic · 小凯 · 2026-05-11

Summary

A June 2025 study by the Qwen team and Tsinghua University's LeapLab re-examines Reinforcement Learning with Verifiable Rewards (RLVR) through the lens of token entropy. Analyzing chain-of-thought reasoning, the authors find that only about 20% of tokens are high-entropy 'forking tokens' that determine which reasoning path is taken, while the remaining 80% are low-entropy tokens that merely execute the chosen path. RLVR training adjusts the probability distributions almost exclusively at these high-entropy decision points. Building on this insight, they propose restricting gradient updates to the top-20% highest-entropy tokens during RLVR. On Qwen3-8B this matches full-gradient training; on Qwen3-14B and Qwen3-32B it surpasses it (AIME'25 +4.79 and +11.04 respectively), while cutting gradient computation by roughly 80%. Qwen3-32B trained this way reaches AIME'24 63.5 (68.1 with 29k context), a sub-600B state of the art, and AIME'25 56.7. Training on low-entropy tokens instead proves actively harmful, and 20% emerges as the optimal ratio across model scales, with benefits growing with size. The paper (arXiv 2506.01939) offers both a mechanistic account of RLVR and a practical recipe for cheaper, more effective LLM reasoning training.

Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective RLVR

> In June 2025, the Qwen team and Tsinghua University's LeapLab jointly published a study re-examining Reinforcement Learning with Verifiable Rewards (RLVR) through the lens of token entropy. The study finds that in chain-of-thought (CoT) reasoning, only ~20% of tokens exhibit high entropy. These "forking tokens" are the key decision points that determine the direction of reasoning. Applying gradient updates only to these 20% of tokens not only matches full-gradient performance on an 8B model but significantly surpasses it on a 32B model (AIME'25 +11.04, AIME'24 +7.71), while cutting training compute by 80%.

1. Background: The Black Box of RLVR

RLVR trains LLM reasoning with outcome-based rewards (e.g., answer correctness) and has driven successes like DeepSeek-R1 and OpenAI o1. Yet its internal learning mechanism remains largely opaque:

What is the model actually "learning"?
Which tokens truly contribute to performance gains?
Are gradient updates evenly distributed across tokens?

Token entropy as an analytical lens

Wang et al. (2025) introduce token-level entropy as an analytical tool:

H(x_t) = -\sum_{v \in \mathcal{V}} P(x_t = v | x_{<t}) \log P(x_t = v | x_{<t})

where $\mathcal{V}$ is the vocabulary and $x_{<t}$ is the preceding context. High entropy means the model is uncertain about the next token; low entropy means it is confident.

2. Core Finding: Identifying Forking Tokens

Systematic analysis of CoT chains reveals a highly skewed entropy distribution:

| Token class | Share | Entropy | Role | |:---|:---:|:---:|:---| | High-entropy forking tokens | ~20% | High | Decision points of reasoning paths | | Low-entropy following tokens | ~80% | Low | Path execution and detail filling |

Forking tokens typically occur at strategy-transition points — e.g., after phrases like "let me try another approach" or "let me verify" — where multiple reasoning strategies diverge.

Tracking entropy before and after RLVR training:

| Stage | High-entropy tokens | Low-entropy tokens | Implication | |:---|:---:|:---:|:---| | Base model | High | Low | Initial distribution | | After RLVR | Adjusted (still high) | Nearly unchanged | RLVR mainly acts on high-entropy tokens |

Key insight: RLVR does not uniformly "teach" the model every token; it precisely reshapes the probability distributions at a minority of critical decision points, enabling better choices at forks.

3. Method: Entropy-Based Gradient Filtering

At each RLVR step:

1. Forward pass generates the full CoT; compute entropy $$H(x_t)$$ per token 2. Select the top $k\%$ highest-entropy tokens ( $$k = 20$$ in experiments) 3. Backpropagate gradients only for selected tokens 4. Gradients for all other tokens are masked to zero

\text{mask}_t = \mathbb{1}[H(x_t) \geq \text{percentile}(\{H(x_i)\}, 100 - k)]

This reduces gradient computation, backward-pass FLOPs, and activation memory by roughly 80%.

4. Experimental Results: Scaling Effect

| Model | Method | AIME'24 | AIME'25 | vs. full gradient | |:---|:---|:---:|:---:|:---| | Qwen3-8B | 20% high-entropy tokens | on par | on par | matches | | Qwen3-14B | 20% high-entropy tokens | +5.21 | +4.79 | surpasses | | Qwen3-32B | 20% high-entropy tokens | +7.71 | +11.04 | significantly surpasses |

Scaling effect: gains from the 20%-token strategy increase monotonically with model size, suggesting larger models better exploit exploration signals from high-entropy tokens.

SOTA results (Qwen3-32B, 20% tokens):

AIME'24: 63.5 (SOTA among models under 600B parameters); 68.1 with context extended to 29k
AIME'25: 56.7

Low-entropy tokens are harmful: training on the 80% low-entropy tokens severely degrades Qwen3-32B AIME'24 performance versus the ~55-56 full-gradient baseline — likely because their gradient noise interferes with learning at key decision points.

Ratio ablation: 10% under-explores and misses key forks; 50% dilutes the signal with low-entropy noise; 20% is the optimal balance across 8B, 14B, and 32B scales.

5. Theoretical Analysis

Decision-tree view: high-entropy tokens correspond to branch nodes; low-entropy tokens correspond to fixed paths within a branch. RLVR's goal is not to make the model confident everywhere, but to make better choices at branch points.
Credit assignment efficiency: in a 1,000-token chain, perhaps only ~20 tokens truly "decide" whether the answer is correct. Optimizing those precisely is more efficient than uniformly updating all 1,000.

6. Relation to Prior Work

| Work | Redundancy found | Level | |:---|:---|:---| | TokenSkip | 40% of CoT tokens can be skipped | Reasoning content | | DAST | Easy questions need less thinking | Compute allocation | | This work | 80% of token gradients can be skipped | Training optimization |

Practical implications for RLVR: enable entropy-based gradient masking at 20% by default; apply more aggressively at larger scales; explore dynamic adjustment of the ratio during training.

7. Limitations and Future Directions

Domain generalization: results are mainly validated on math (AIME). Code generation may have more forks; scientific deduction may have fewer; multimodal decision points are undefined.
Better fork metrics: conditional entropy change rate, mutual information with the answer, or attention concentration may outperform raw entropy.
Exploration-exploitation: high-entropy tokens are exploration points; this work offers a new operationalization of balancing exploration and exploitation in RL.

8. Conclusion

Beyond the 80/20 Rule reveals the deep structure of RLVR training: reasoning gains come not from uniform optimization of all tokens, but from precise adjustment of a minority of high-entropy forking tokens. The finding has both scientific value (a new mechanistic perspective on RLVR) and engineering value (an 80% reduction in RL training cost). As LLM training costs climb, "train fewer tokens, but the right tokens" may become a core principle of next-generation efficient RL algorithms.

Paper Details

| Item | Content | |:---|:---| | Title | Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning | | Authors | Shenzhi Wang, Le Yu, Chang Gao, Chujie Zheng, Shixuan Liu, Rui Lu, Kai Dang, Xionghui Chen, Jianxin Yang, Zhenru Zhang, Yuqiong Liu, An Yang, Andrew Zhao, Yang Yue, Shiji Song, Bowen Yu, Gao Huang, Junyang Lin | | Affiliations | Qwen Team (Alibaba Inc.), LeapLab (Tsinghua University) | | arXiv ID | 2506.01939 | | Date | 2025-06-02 | | Project page | https://shenzhi-wang.github.io/high-entropy-minority-tokens-rlvr |