Token Entropy vs Attention Entropy: Two Papers Find '20% of Tokens Suffice'—But Define Critical Tokens in Opposite Ways

> Core judgment: Li et al. (2026) revisit token-level RL learning signals through the lens of attention entropy, producing a picture that both echoes and contradicts Round 14's "80/20 Rule." Both find that a 20% token subset retains most performance—but Round 14 says high token-entropy (prediction-uncertain) tokens are key, while this paper argues low-attention-entropy (context-focused) tokens are the "anchors" providing a stable optimization skeleton, and high-attention-entropy tokens are "explorers": volatile, yet possibly containing hard-reasoning signal. Crucially, the paper proposes dynamic entropy-aware soft reweighting, lifting Qwen3-8B from 34.39 to 37.40. If this is right, token selection is not "one-size-fits-all"—it requires dynamically balancing anchors and explorers by training phase and task difficulty.

1. Two camps: both say 20% suffices, but which 20%?

1.1 Round 14's "80/20 Rule"

Wang et al. (Round 14) analyze via token entropy (entropy of the next-token prediction distribution):

| Token type | Definition | Role | Selection strategy | |:---|:---|:---|:---| | High token entropy | Scattered prediction distribution | Branching/decision points | Keep this 20% | | Low token entropy | Concentrated prediction distribution | Follower/filler tokens | Mask out |

> Result: training only on the top-20% high token-entropy tokens gave a 32B model +11.04 on AIME'25.

1.2 This paper's "Anchor–Explorer" spectrum

Li et al. use attention entropy (how concentrated vs. dispersed the attention over context is):

| Token type | Definition | Role | Gradient behavior | |:---|:---|:---|:---| | Low attention entropy — Anchors | Concentrated contextual support | Stable skeleton | Stable, aligned with full-gradient direction | | High attention entropy — Explorers | Dispersed contextual support | Hard-reasoning signal | Large but volatile and unstable |

> Key findings: > - Anchors-only training = stable but plateaus (stalls on hard tasks) > - Explorers-only training = unstable on average, but a few successful runs show hard-reasoning breakthroughs > - Both are necessary, but must be dynamically balanced

1.3 Comparing the two entropies

| Dimension | Token entropy (Round 14) | Attention entropy (this paper) | |:---|:---|:---| | Measures | Model's prediction uncertainty for the next token | How dispersed attention is over context | | High entropy means | "Many possible next tokens" | "Relies on broad contextual information" | | Low entropy means | "Next token nearly certain" | "Relies on concentrated local information" | | Critical tokens | High token entropy = decision points | Low attention entropy = stable anchors | | 20% strategy | Keep high token entropy | Both needed, dynamically balanced |

> Not a contradiction but a complement: token entropy tells the model *where decisions happen*; attention entropy tells it *how to optimize stably*.

2. Anchors: the model's safety net

Low-attention-entropy tokens:

• Rely on concentrated contextual support (e.g., a number or formula just mentioned)
• Produce stable gradients aligned with the full-token update direction
• Form the reliable backbone of optimization

Anchors-only training is highly stable and converges fast, works well on easy tasks, but plateaus on hard tasks—anchors provide "safe" signals, so the model follows known patterns without incentive to break out.

3. Explorers: the model's adventurers

High-attention-entropy tokens:

• Rely on dispersed contextual support (synthesizing multiple distant pieces of information)
• Produce larger but more volatile gradients
• May contain hard-reasoning signal

Explorers-only training is unstable on average and can oscillate or overfit on easy tasks, but a minority of successful runs show breakthroughs on hard reasoning—explorers do contain paths to hard reasoning; the optimization is just too volatile for most runs.

4. Dynamic balance: entropy-aware soft reweighting

The insight: don't choose anchors *or* explorers—dynamically adjust their weights:

| Phase / scenario | Anchor weight | Explorer weight | Rationale | |:---|:---:|:---:|:---| | Early training | High | Low | Build a stable foundation | | Mid training | Medium | Medium | Gradually introduce exploration | | Late training | Low | High | Break through plateaus | | Easy tasks | High | Low | No need to take risks | | Hard tasks | Low | High | Hard reasoning needed |

Results on Qwen3-8B-Base (held-out average):

| Configuration | Score | |:---|:---:| | Baseline (all tokens) | 34.39 | | Dynamic entropy-aware reweighting | 37.40 | | Gain | +3.01 |

> The +3.01 comes not from more compute, but from smarter token weight allocation—anchors stabilize training while explorers supply breakthrough signals at key moments.

5. Deeper dialogue with Round 14

| Round 14 | This paper | Unified view | |:---|:---|:---| | High token entropy is critical | Low-attention-entropy anchors stabilize training | Each measures a different dimension of "criticality" | | Use only top-20% high-entropy tokens | 20% subset suffices, but needs dynamic combination | 20% is the sparsity threshold; the combination strategy differs | | 32B +11 AIME'25 | Qwen3-8B +3 held-out | Different scales and metrics, but both work |

Token entropy: \(H_{\text{token}} = -\sum_v p(v|x_{<t}) \log p(v|x_{<t})\) — is this position a decision point?

Attention entropy: \(H_{\text{attention}} = -\sum_i \alpha_i \log \alpha_i\), where \(\alpha_i\) are attention weights — does this token rely on concentrated local information (anchor) or dispersed global information (explorer)?

Combined "identity card":

| | Low token entropy | High token entropy | |:---:|:---:|:---:| | Low attention entropy | Certain + local = follower token | Uncertain + local = branching token (Round 14) | | High attention entropy | Certain + global = synthesis token | Uncertain + global = explorer (this paper) |

6. My bet

I bet $1,000: by the end of 2026, "dual-entropy dynamic balancing" will be standard in RL training. All major frameworks will monitor both token entropy and attention entropy, adjusting token weights dynamically by training phase.

Why: 1. Both entropies reveal real structure—together they characterize token roles more completely than either alone. 2. Dynamic balancing beats fixed selection—Round 14's fixed 20% works under specific conditions; dynamic adjustment goes further. 3. Low implementation cost—attention entropy is essentially free in the transformer forward pass (it's the attention weights). 4. Theoretically elegant—upgrades token selection from "one-size-fits-all" to "context-aware." 5. Compatible with all RL algorithms—GRPO, PPO, DAPO can all add dual-entropy reweighting.

7. Limitations and future work

• Interaction between the two entropies: their joint distribution and special roles of "double-high" or "double-low" tokens are unexplored.
• Layer-wise differences: do shallow vs. deep explorers serve different functions?
• Head-wise differences: are there attention heads specialized for "exploration"?
• Integration with uncertainty profiles: can dual-entropy analysis extend to the trajectory level ("anchor-dominated" vs. "explorer-dominated" trajectories)?

Ultimately, this paper and Round 14 together offer a dual perspective on token-level RL—not "who is right," but "both are right, each seeing a different facet."

Paper details

| Item | Content | |:---|:---| | Title | Not All Tokens Learn Alike: Attention Entropy Reveals Heterogeneous Signals in RL Reasoning | | Authors | Gengyang Li, Zheng-Fan Wu, Siqi Bao, Yunfang Wu | | arXiv ID | 2605.07660 | | Date | 2026-05-08 | | Contributions | Attention-entropy analysis of token-level RL heterogeneity; Anchors vs. Explorers spectrum; 20% subset sparsity; dynamic entropy-aware soft reweighting; Qwen3-8B-Base 34.39→37.40 | | Key results | Uniform random 20% retains most performance; anchors stable but plateau; explorers volatile but carry hard-reasoning signal; dynamic reweighting +3.01 held-out average |