Not All Tokens Learn Alike: Attention Entropy Reveals Heterogeneous Learning Signals in RL Reasoning

Forum topic · 小凯 · 2026-05-11

Summary

A May 2026 paper by Li et al. examines token-level heterogeneity in LLM post-training RL through the lens of attention entropy. The study confirms that token-level RL objectives are sparsely estimable: a uniform random 20% token subset retains most of the full-token held-out performance. However, entropy-structured subsets behave very differently. Low attention-entropy "anchor" tokens concentrate attention on few context positions, produce stable gradients aligned with full-token updates, and provide an optimization backbone, but plateau on hard benchmarks. High attention-entropy "explorer" tokens aggregate broadly distributed context, yield larger but volatile gradients, and may carry hard-reasoning breakthrough signals despite unstable training. Control experiments rule out position bias, prediction-entropy confounds, and loss normalization artifacts. Building on these findings, the authors propose a dynamic entropy-aware soft reweighting scheme that adjusts anchor/explorer contributions by training stage and task difficulty, improving Qwen3-8B-Base held-out average performance from 34.39 to 37.40 (+3.01). The work complements token-entropy research such as the "Beyond the 80/20 Rule" line, suggesting token importance is jointly determined by predictive uncertainty and contextual dependency patterns.

Not All Tokens Learn Alike: Attention Entropy Reveals Heterogeneous Signals in RL Reasoning

> May 2026 — Li et al. study the heterogeneity of token-level learning signals in LLM post-training RL through the lens of attention entropy. Token-level RL objectives are sparsely estimable — a uniform random 20% token subset retains most full-token held-out performance — but entropy-structured subsets behave very differently. Low attention-entropy "anchor" tokens provide a stable optimization backbone, while high attention-entropy "explorer" tokens produce volatile gradients yet may contain hard-reasoning breakthrough signals. Based on this, the authors propose a dynamic entropy-aware soft reweighting intervention that improves Qwen3-8B-Base held-out average performance from 34.39 to 37.40.

---

1. Background: Understanding Token-Level RL Signals

Prior work (e.g., "Beyond the 80/20 Rule") found token-level gradient updates are highly sparse: only ~20% of tokens are needed to retain most performance. But studies disagree on *which* 20%:

| Study | Metric | Key token characteristic | Strategy | |:---|:---|:---|:---| | Prior 80/20 work | Token entropy | High predictive uncertainty | Keep high-entropy tokens | | This paper | Attention entropy | Low / high attention concentration | Dynamically balance anchors and explorers |

Attention entropy definition

For the $$t$$ -th token in a response, attention entropy measures how concentrated or diffuse its contextual support is when generated:

H_{\text{attn}}^{(t)} = -\sum_{i} \alpha_i^{(t)} \log \alpha_i^{(t)}

where $\alpha_i^{(t)}$ is the attention weight on the $$i$$ -th context position when generating that token.

Low attention entropy: the model relies on a few key context positions (concentrated support)
High attention entropy: the model integrates broadly distributed context (diffuse support)

---

2. The Anchor–Explorer Spectrum

Anchors (low attention entropy)

| Property | Characteristic | |:---|:---| | Context dependency | Concentrated (few high-weight positions) | | Gradient behavior | Stable, aligned with full-token updates | | Optimization role | Reliable backbone, training stability | | Limitation | Tends to plateau on hard benchmarks |

Explorers (high attention entropy)

| Property | Characteristic | |:---|:---| | Context dependency | Diffuse (broad attention distribution) | | Gradient behavior | Larger but more volatile | | Optimization role | May carry hard-reasoning breakthrough signals | | Limitation | Unstable on average; success is stochastic |

Controlled comparison

| Configuration | Stability | Easy tasks | Hard tasks | Interpretation | |:---|:---:|:---:|:---:|:---| | Anchors-only | High | Good | Plateau | Lacks breakthrough momentum | | Explorers-only | Low | Unstable | Few breakthroughs | Strong but noisy signal | | Dynamic balance | Mid-high | Good | Breakthroughs | Best combination |

> Key finding: Explorers-only training is unstable on average, but "a few successful runs" show significant performance breakthroughs, confirming these tokens do contain valuable hard-reasoning signals.

---

3. Sparsity Validation

Uniform random subsets: a 20% uniform random subset retains most held-out performance, confirming the redundancy of token-level updates.
Entropy-structured subsets:
Low attention entropy (anchors): stable convergence but limited ceiling
High attention entropy (explorers): high variance but potentially higher ceiling

> Core insight: Sparsity doesn't just mean "you can update fewer tokens" — it means different token types serve different optimization functions.

---

4. Ruling Out Alternative Explanations

Control experiments confirm the observed asymmetry is not caused by:

| Alternative hypothesis | Control | Conclusion | |:---|:---|:---:| | Position bias | Controlled token position | ❌ Not a position effect | | Prediction-entropy confound | Distinguished token entropy from attention entropy | ❌ Not predictive uncertainty | | Loss normalization | Controlled loss scaling | ❌ Not a normalization artifact |

> The anchor–explorer asymmetry is an intrinsic phenomenon tied to attention structure.

---

5. Dynamic Entropy-Aware Soft Reweighting

Rather than fixed selection, token weights adapt to training dynamics:

w_t^{(i)} = f(H_{\text{attn}}^{(i)}, \text{training\_stage}, \text{task\_difficulty})

| Factor | Anchor weight | Explorer weight | |:---|:---:|:---:| | Early training | ↑ | ↓ | | Late training | ↓ | ↑ | | Easy tasks | ↑ | ↓ | | Hard tasks | ↓ | ↑ |

Results

| Configuration | Qwen3-8B-Base held-out avg | |:---:|:---:| | Baseline (full tokens) | 34.39 | | Dynamic entropy-aware reweighting | 37.40 | | Absolute gain | +3.01 |

> The gain comes not from more compute, but smarter token weight allocation — lean on anchors for stability, activate explorers for breakthroughs.

---

6. Relation to Related Work

vs. the 80/20 rule (token entropy): token entropy captures decision points; attention entropy captures contextual dependency. A unified view: high token entropy + low attention entropy = "branching anchor"; high + high = "global explorer"; low + low = "local filler"; low + high = "global synthesis".
vs. POISE-style value signals: attention-entropy statistics may carry richer value information — predicting not just answer correctness but whether a token stabilizes training or takes exploratory risk.
vs. trajectory uncertainty tracing: the anchor–explorer spectrum can be viewed as token-level "uncertainty identity" along a trajectory — each position is either a producer (explorer) or consumer (anchor) of uncertainty.

---

7. Limitations and Future Directions

1. Layer-wise differences: shallow layers may host more anchors (local pattern matching); deep layers more explorers (global semantic integration) 2. Head specialization: do "anchor heads" and "explorer heads" exist functionally? 3. Task structure: math reasoning may be anchor-heavy; open-ended generation explorer-heavy 4. Adaptive reweighting: online loss-curvature monitoring, closed-loop validation feedback, joint optimization with RL rewards

---

8. Conclusion

Core contributions:

1. Dual-role identification: tokens divide into functional types — stable anchors vs. exploratory signals 2. Sparsity reconfirmed: 20% subsets suffice, but *which* 20% should be context-adaptive 3. Dynamic balancing: adjust anchor–explorer weights by training stage and task difficulty 4. Empirical gain: Qwen3-8B-Base +3.01 held-out average

Combined with token-entropy research, this work suggests token importance is not one-dimensional: it is jointly determined by predictive uncertainty (token entropy) and contextual dependency patterns (attention entropy). Future efficient RL training may need to monitor both entropy signals and adapt optimization accordingly.

---

Paper Details

| Item | Content | |:---|:---| | Title | Not All Tokens Learn Alike: Attention Entropy Reveals Heterogeneous Signals in RL Reasoning | | Authors | Gengyang Li, Zheng-Fan Wu, Siqi Bao, Yunfang Wu | | arXiv ID | 2605.07660 | | Date | 2026-05-08 | | Key results | Uniform random 20% retains most performance; anchors stable but plateau; explorers volatile but contain hard-reasoning signal; dynamic reweighting +3.01 on Qwen3-8B-Base |