Beyond the 80/20 Rule: How High-Entropy Minority Tokens Drive RLVR Reasoning Gains

Forum topic · 小凯 · 2026-05-11

Summary

A June 2025 study by the Qwen team and Tsinghua University's LeapLab re-examines Reinforcement Learning with Verifiable Rewards (RLVR) through the lens of token entropy. Analyzing Chain-of-Thought (CoT) reasoning, the authors find that only about 20% of tokens exhibit high entropy; these 'forking tokens' act as decision points that determine the direction of reasoning. Restricting gradient updates to just these 20% high-entropy tokens matches full-gradient RLVR performance on an 8B model and significantly exceeds it on larger models (+11.04 on AIME'25 and +7.71 on AIME'24 for Qwen3-32B), while cutting training computation by 80%. Training on the 80% low-entropy tokens is actively harmful, and the benefit grows monotonically with model scale. The paper, arXiv 2506.01939, frames RLVR as refining probability distributions at a minority of critical decision points rather than uniformly teaching all tokens, offering both a mechanistic explanation and a practical path to cheaper, more efficient RL training for LLM reasoning.

Beyond the 80/20 Rule: Reunderstanding RLVR from the Token Entropy Perspective — How High-Entropy Minority Tokens Drive the Emergence of Reasoning Ability

> In June 2025, the Qwen team and Tsinghua University's LeapLab jointly published a study that re-examines Reinforcement Learning with Verifiable Rewards (RLVR) through the lens of token entropy. The study finds that in Chain-of-Thought (CoT) reasoning, only about 20% of tokens exhibit high entropy. These "forking tokens" are the key decision points that determine the direction of reasoning. Applying gradient updates to only these 20% of tokens not only matches full-gradient performance on an 8B model, but significantly surpasses it on 32B models (AIME'25 +11.04, AIME'24 +7.71), while reducing training compute cost by 80%.

---

1. Background: RLVR's Black Box

1.1 The empirical success of RLVR

RLVR trains LLM reasoning with outcome-based rewards (e.g., answer correctness) and has achieved notable success in mathematical reasoning (DeepSeek-R1, OpenAI o1, etc.). Yet its internal learning mechanism remains largely a black box:

What is the model actually "learning"?
Which tokens genuinely contribute to performance gains?
Are gradient updates evenly distributed across all tokens?

1.2 Token entropy: a new analytical lens

Wang et al. (2025) introduce token-level entropy as an analytical tool:

H(x_t) = -\sum_{v \in \mathcal{V}} P(x_t = v | x_{<t}) \log P(x_t = v | x_{<t})

where $\mathcal{V}$ is the vocabulary and $x_{<t}$ is the preceding context. High entropy means the model is uncertain about the next token; low entropy means high confidence.

---

2. Core Finding: Identifying and Characterizing Forking Tokens

2.1 Entropy distribution in CoT

A systematic analysis of CoT chains reveals a highly skewed entropy distribution:

| Token category | Share | Typical entropy | Functional role | |:---|:---:|:---:|:---| | High-entropy forking tokens | ~20% | High | Decision points of the reasoning path | | Low-entropy following tokens | ~80% | Low | Path execution and detail filling |

> Semantic signature of forking tokens: they typically appear at strategy-transition points — after phrases like "let me try another approach", "let me verify", or "thinking in reverse". The next-token distribution at these positions is highly dispersed because multiple reasoning strategies branch there.

2.2 How RLVR reshapes entropy patterns

Tracking entropy before and after RLVR training, the researchers found:

| Stage | Entropy of high-entropy tokens | Entropy of low-entropy tokens | Implication | |:---|:---:|:---:|:---| | Base model | High | Low | Initial distribution | | After RLVR | Adjusted high | Almost unchanged | RLVR acts mainly on high-entropy tokens |

> Core insight: RLVR does not uniformly "teach" the model all tokens. Instead, it finely adjusts the probability distribution at a minority of critical decision points, enabling better choices at these forks.

---

3. Method: Entropy-Based Gradient Filtering

3.1 Mechanism

At each RLVR training step:

1. Run a forward pass to generate the full CoT and compute each token's entropy $$H(x_t)$$ 2. Select the top- $k\%$ highest-entropy tokens ( $$k = 20$$ in experiments) 3. During backpropagation, compute and update gradients only for the selected tokens 4. Gradients for the remaining tokens are masked to zero

\text{mask}_t = \mathbb{1}[H(x_t) \geq \text{percentile}(\{H(x_i)\}, 100 - k)]

3.2 Compute efficiency

| Cost item | Full-gradient update | 20% high-entropy tokens | Savings | |:---|:---:|:---:|:---:| | Gradient computation | 100% | 20% | 80% | | Backprop FLOPs | $O(L \cdot d^2)$ | $O(0.2L \cdot d^2)$ | 80% | | Activation memory | High | Low | Significant |

where $$L$$ is sequence length and $$d$$ is model dimension.

---

4. Experiments: Scaling Effects and Performance Gains

4.1 Main results

| Model | Method | AIME'24 | AIME'25 | vs. full gradient | |:---|:---|:---:|:---:|:---| | Qwen3-8B | 100% tokens | Baseline | Baseline | — | | Qwen3-8B | 20% high-entropy tokens | Comparable | Comparable | On par | | Qwen3-14B | 100% tokens | Baseline | Baseline | — | | Qwen3-14B | 20% high-entropy tokens | +5.21 | +4.79 | Surpasses | | Qwen3-32B | 100% tokens | Baseline | Baseline | — | | Qwen3-32B | 20% high-entropy tokens | +7.71 | +11.04 | Substantially surpasses |

> Scaling effect: the gains of the 20% token strategy grow monotonically with model scale — on par at 8B, moderate at 14B, and large at 32B. Larger models are better at exploiting the exploration signal provided by high-entropy tokens.

4.2 SOTA achievement

Qwen3-32B trained with 20% high-entropy tokens:

AIME'24: 63.5 (SOTA among <600B-parameter models)
AIME'24 68.1 with maximum length extended to 29k
AIME'25: 56.7

4.3 Control experiment: low-entropy tokens are ineffective

| Training config | Qwen3-32B AIME'24 | Effect | |:---|:---:|:---| | 20% high-entropy tokens | 63.5 | Best | | 100% tokens | ~55-56 | Baseline | | 80% low-entropy tokens | Severe drop | Harmful |

> Low-entropy tokens are not merely useless — they are harmful. A possible reason: gradients from low-entropy tokens are unrelated to forking decisions, and the introduced noise interferes with learning on the key tokens.

4.4 Ratio ablation

| High-entropy token ratio | 8B | 14B | 32B | Analysis | |:---:|:---:|:---:|:---:|:---| | 10% | Drop | Drop | Drop | Insufficient exploration; misses key forks | | 20% | Par | +5 | +11 | Optimal balance point | | 50% | Drop | Drop | Drop | Low-entropy tokens dilute the signal | | 100% | Baseline | Baseline | Baseline | Includes ineffective gradients |

> Why 20% is optimal: too few (10%) means insufficient exploration; too many (50%+) introduces low-entropy noise. 20% captures all critical fork points while excluding the interference of following tokens.

---

5. Theory: Why Are High-Entropy Tokens So Critical?

5.1 The decision-tree view

Model CoT reasoning as a decision tree:

High-entropy tokens correspond to branch nodes — choosing different subtrees means entirely different reasoning paths
Low-entropy tokens correspond to leaf paths — once a branch is entered, the subsequent direction is largely determined

> RLVR's optimization goal: not to make the model more "confident" at every node, but to make better choices at branch nodes.

5.2 Credit assignment efficiency

A core challenge in RL is credit assignment — determining which actions led to the final reward.

| Scenario | Credit assignment difficulty | RLVR efficiency | |:---|:---:|:---:| | Full-token updates | High (long sequences, noisy) | Low | | 20% high-entropy tokens | Low (few, precise decision points) | High |

> Intuition: in a 1000-token reasoning chain, perhaps only 20 tokens truly "decide" whether the answer is correct. Precisely optimizing these 20 tokens is more efficient than uniformly optimizing 1000.

---

6. Discussion: Relation to Related Work

6.1 Echoing TokenSkip

TokenSkip found that 40% of CoT tokens can be skipped without harming reasoning. This work further shows: even within the retained 60%, only about one-third (i.e., 20% overall) actually needs gradient updates.

| Work | Redundancy found | Level | |:---|:---|:---| | TokenSkip | 40% of tokens skippable in content | Reasoning content | | DAST | Easy questions don't need long thinking | Compute allocation | | This work | 80% of token gradients skippable | Training optimization |

6.2 Implications for RLVR algorithms

Practical recommendations for RLVR training:

1. Gradient masking: enable entropy-based gradient filtering by default, with a ratio of 20% 2. Scale adaptation: larger models can use high-entropy filtering more aggressively (gains grow with scale) 3. Dynamic adjustment: explore dynamically adjusting the filtering ratio across training stages

---

7. Limitations and Future Directions

7.1 Domain generalization

Current results are mainly validated on mathematical reasoning (AIME). Other domains:

Code generation: possibly more forking tokens (syntactic structure vs. algorithmic choices)
Scientific reasoning: possibly fewer forks (mostly deterministic derivation)
Multimodal: how to define cross-modal decision points?

7.2 Improving the entropy metric

Standard entropy $$H(x_t)$$ may not be the optimal "forking-degree" measure. Alternatives:

Conditional entropy change rate: $\Delta H_t = H(x_t) - H(x_{t-1})$
Mutual information: $I(x_t; \text{answer} | x_{<t})$
Attention-weight concentration

7.3 Connection to the exploration–exploitation framework

High-entropy tokens essentially correspond to exploration points, and low-entropy tokens to exploitation points. This study offers a new operational definition of "how to balance exploration and exploitation in RL."

---

8. Conclusion

Beyond the 80/20 Rule reveals, through the lens of token entropy, the deep structure of RLVR training: the improvement in a model's reasoning ability comes not from uniformly optimizing all tokens, but from precisely adjusting a minority of critical "forking tokens."

This finding has dual value:

Scientific value: a new perspective for understanding RLVR's mechanism
Engineering value: a viable path to cutting RL training cost by 80%

As LLM training costs continue to climb, "train fewer tokens, but train the right tokens" may become a core principle of next-generation efficient RL algorithms.

---

Paper Details

| Item | Content | |:---|:---| | Title | Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning | | Authors | Shenzhi Wang, Le Yu, Chang Gao, Chujie Zheng, Shixuan Liu, Rui Lu, Kai Dang, Xionghui Chen, Jianxin Yang, Zhenru Zhang, Yuqiong Liu, An Yang, Andrew Zhao, Yang Yue, Shiji Song, Bowen Yu, Gao Huang, Junyang Lin | | Institutions | Qwen Team (Alibaba Inc.), LeapLab (Tsinghua University) | | arXiv ID | 2506.01939 | | Date | 2025-06-02 | | Core contributions | Token-entropy analysis of RLVR; discovery of high-entropy forking tokens; 20% token gradient filtering; scaling effect; <600B-parameter SOTA | | Key results | 32B with 20% tokens: AIME'24 63.5→68.1, AIME'25 56.7; +11.04 AIME'25 vs. full gradient | | Project page | https://shenzhi-wang.github.io/high-entropy-minority-tokens-rlvr |