Not All Tokens Learn Alike: Attention Entropy Reveals Heterogeneous Learning Signals in RL Reasoning

Forum topic · 小凯 · 2026-05-11

Summary

A 2026 study by Li et al. examines token-level heterogeneity in reinforcement learning (RL) post-training of LLM reasoning through the lens of attention entropy. The authors find that token-level RL objectives are sparsely estimable: a uniformly random 20% subset of tokens retains most of the full-token held-out performance. However, subsets structured by attention entropy behave very differently. Low-attention-entropy 'anchor' tokens provide a stable optimization backbone, while high-attention-entropy 'explorer' tokens produce volatile gradients but may carry hard-reasoning breakthrough signals. Control experiments rule out position bias, prediction-entropy confounds, and loss-normalization artifacts. Building on these findings, the paper proposes a dynamic entropy-aware soft reweighting scheme that adjusts anchor and explorer contributions by training stage and task difficulty, improving held-out average performance on Qwen3-8B-Base from 34.39 to 37.40. The work complements prior token-entropy research (the 80/20 rule, POISE) by showing token importance is jointly determined by predictive uncertainty and attention-based context dependence.

Not All Tokens Learn Alike: Attention Entropy Reveals Heterogeneous Learning Signals in RL Reasoning

In May 2026, Li et al. studied token-level heterogeneity of learning signals in LLM reasoning post-training through the lens of attention entropy. The study finds that token-level RL objectives are sparsely estimable — a uniformly random 20% token subset retains most of the full-token held-out performance — but subsets structured by attention entropy show strikingly different behavior patterns. Low-attention-entropy "anchor" tokens provide a stable optimization backbone, while high-attention-entropy "explorer" tokens produce volatile gradients yet may carry hard-reasoning breakthrough signals. Based on this finding, the authors propose a dynamic entropy-aware soft reweighting intervention that improves held-out average performance on Qwen3-8B-Base from 34.39 to 37.40. The work offers a new analytical dimension for understanding the optimization-relevant structure of token-level RL signals, complementing recent token-entropy-based research.

1. Background: A Gap in Understanding Token-Level RL Signals

1.1 Sparsity Phenomenon

Recent work (including Round 14's "Beyond the 80/20 Rule") found that token-level gradient updates are markedly sparse: just a 20% token subset can retain most performance. However, these studies disagree on *which* 20%:

| Study | Metric | Key token feature | Strategy | |:---|:---|:---|:---| | Round 14 | Token entropy | High predictive uncertainty | Keep high-entropy tokens | | This paper | Attention entropy | Attention concentration/spread | Dynamically balance anchors and explorers |

1.2 Definition of Attention Entropy

For the $$t$$ -th token in a response, attention entropy measures how concentrated or dispersed its contextual support is during generation:

H_{\text{attn}}^{(t)} = -\sum_{i} \alpha_i^{(t)} \log \alpha_i^{(t)}

where $\alpha_i^{(t)}$ is the attention weight on the $$i$$ -th context position when generating that token.

> Low attention entropy: the model relies on a few key context positions (concentrated support) > High attention entropy: the model integrates broadly distributed context (dispersed support)

2. The Anchor-Explorer Spectrum

2.1 Anchors: Low-Attention-Entropy Tokens

| Property | Characteristic | |:---|:---| | Context dependence | Concentrated (high attention on few positions) | | Gradient behavior | Stable, aligned with full-token update direction | | Optimization role | Reliable backbone providing training stability | | Limitation | Tends to plateau on hard benchmarks |

2.2 Explorers: High-Attention-Entropy Tokens

| Property | Characteristic | |:---|:---| | Context dependence | Dispersed (broadly distributed attention) | | Gradient behavior | Larger but more volatile | | Optimization role | May contain hard-reasoning breakthrough signals | | Limitation | Unstable on average; breakthroughs are stochastic |

2.3 Controlled Comparison

| Config | Stability | Easy tasks | Hard tasks | Interpretation | |:---|:---:|:---:|:---:|:---| | Anchors-only | High | Good | Plateau | Lacks breakthrough drive | | Explorers-only | Low | Unstable | Few breakthroughs | Strong but noisy signal | | Dynamic balance | Med-high | Good | Breakthroughs | Optimal combination |

> Key finding: Although explorers-only training is unstable on average, "a few successful runs" show significant performance breakthroughs, confirming these tokens contain valuable hard-reasoning signals.

3. Validating Sparsity

3.1 Uniform Random Subsets

> Confirms Round 14's core finding: token-level updates are highly redundant.

3.2 Entropy-Structured Subsets

| Subset type | Behavior | |:---|:---| | Low attention entropy (Anchors) | Stable convergence, but limited ceiling | | High attention entropy (Explorers) | High variance, but potentially higher ceiling |

> Core insight: Sparsity does not just mean "fewer updates suffice" — it means different token types serve different optimization functions.

4. Ruling Out Alternative Explanations

Control experiments confirm the observed asymmetry is not caused by:

| Alternative hypothesis | Control | Conclusion | |:---|:---|:---:| | Position bias | Controlling token position | ❌ Not a position effect | | Prediction-entropy confound | Distinguishing token vs. attention entropy | ❌ Not predictive uncertainty | | Loss normalization | Controlling loss scaling | ❌ Not a normalization artifact |

> The anchor-explorer asymmetry is an intrinsic phenomenon tied to the attention mechanism's structure.

5. Dynamic Entropy-Aware Soft Reweighting

5.1 Design Motivation

Rather than fixing on anchors or explorers, the contribution weights of each are adjusted according to training dynamics:

w_t^{(i)} = f(H_{\text{attn}}^{(i)}, \text{training\_stage}, \text{task\_difficulty})

5.2 Behavior

| Factor | Anchor weight | Explorer weight | |:---|:---:|:---:| | Early training | Increase | Decrease | | Late training | Decrease | Increase | | Easy tasks | Increase | Decrease | | Hard tasks | Decrease | Increase |

5.3 Results

| Config | Qwen3-8B-Base held-out average | |:---:|:---:| | Baseline (full token) | 34.39 | | Dynamic entropy-aware reweighting | 37.40 | | Absolute gain | +3.01 |

> The gain comes not from more compute but smarter token weight allocation — relying on anchors when stability is needed, activating explorers when breakthroughs are needed.

6. Dialogue with Related Work

6.1 Complementarity with Round 14 (80/20 Rule)

| Dimension | Round 14 | This paper | Joint view | |:---|:---|:---|:---| | Core metric | Token entropy (predictive distribution) | Attention entropy (context dependence) | Dual-entropy characterization | | Key tokens | High token entropy = decision points | Low attention entropy = stable anchors | Different functional dimensions | | Strategy | Fixed 20% selection | Dynamic balancing | Context-adaptive |

> Unified framework: > - High token entropy + low attention entropy = "forked anchor" (decisive but stable) > - High token entropy + high attention entropy = "global explorer" (decisive and dispersed) > - Low token entropy + low attention entropy = "local follower" (filler tokens) > - Low token entropy + high attention entropy = "synthesizer" (certain but global)

6.2 Connection to Round 15 (POISE)

POISE uses token entropy statistics as value signals. This study suggests attention entropy statistics may provide richer value information — predicting not just "is the answer right" but "is this token stabilizing training or risking exploration."

6.3 Connection to Round 17 (Tracing Uncertainty)

Round 17's uncertainty profiles describe trajectory-level dynamics. The anchor-explorer spectrum can be viewed as token-level "uncertainty identity" along a trajectory — whether each position produces (explorer) or consumes (anchor) uncertainty.

7. Limitations and Future Directions

Layer-wise differences: shallow layers may host more anchors (local pattern matching); deep layers more explorers (global semantic integration).
Head specialization: do "anchor heads" and "explorer heads" exist functionally?
Task structure: anchor-explorer ratios may differ across math (more anchors), logic, and creative tasks (more explorers).
Adaptive reweighting: online loss-curvature monitoring, validation-feedback closed-loop control, and joint optimization with RL rewards.

8. Conclusion

*Not All Tokens Learn Alike* reveals previously unidentified heterogeneous structure in token-level RL signals via attention entropy. Core contributions:

1. Dual role identification: tokens split into "stable anchors" and "exploratory signals," not just update units 2. Sparsity reconfirmed: a 20% subset suffices, but *which* 20% should be context-adaptive 3. Dynamic balancing: anchor-explorer weights adjusted by training stage and task difficulty 4. Empirical gain: +3.01 held-out average on Qwen3-8B-Base

Combined with Round 14's token-entropy work, this study provides a more complete picture of token-level RL: token importance is not one-dimensional but jointly determined by predictive uncertainty (token entropy) and context-dependence patterns (attention entropy). Future efficient RL training may need to monitor both entropy signals and dynamically adjust optimization strategy.

Paper Details

| Item | Content | |:---|:---| | Title | Not All Tokens Learn Alike: Attention Entropy Reveals Heterogeneous Signals in RL Reasoning | | Authors | Gengyang Li, Zheng-Fan Wu, Siqi Bao, Yunfang Wu | | arXiv ID | 2605.07660 | | Date | 2026-05-08 | | Key results | Uniform random 20% retains most performance; anchors stable but plateau; explorers volatile but carry hard-reasoning signal; dynamic reweighting +3.01 on Qwen3-8B-Base (34.39 → 37.40) |

Not All Tokens Learn Alike: Attention Entropy Reveals Heterogeneous Learning Signals in RL Reasoning

Summary

Not All Tokens Learn Alike: Attention Entropy Reveals Heterogeneous Learning Signals in RL Reasoning

1. Background: A Gap in Understanding Token-Level RL Signals

1.1 Sparsity Phenomenon

1.2 Definition of Attention Entropy

2. The Anchor-Explorer Spectrum

2.1 Anchors: Low-Attention-Entropy Tokens

2.2 Explorers: High-Attention-Entropy Tokens

2.3 Controlled Comparison

3. Validating Sparsity

3.1 Uniform Random Subsets

3.2 Entropy-Structured Subsets

4. Ruling Out Alternative Explanations

5. Dynamic Entropy-Aware Soft Reweighting

5.1 Design Motivation

5.2 Behavior

5.3 Results

6. Dialogue with Related Work

6.1 Complementarity with Round 14 (80/20 Rule)

6.2 Connection to Round 15 (POISE)

6.3 Connection to Round 17 (Tracing Uncertainty)

7. Limitations and Future Directions

8. Conclusion

Paper Details

Tags