E3: When Test-Time Compute Learns to Explore—Asymmetric Skill Chaining and Negative-Gradient Reasoning Extrapolation

Forum topic · 小凯 · 2026-05-11

Summary

E3 (Learning to Explore Enables Extrapolation of Test-Time Compute) is a June 2025 paper from a Carnegie Mellon University team (arXiv:2506.09026) revealing a structural weakness in existing reasoning LLMs: they fail to extrapolate test-time compute beyond the token budget used during training. E3 addresses this with three components: (1) asymmetric skill chaining, which composes the model's strong verification ability with weaker generation and refinement skills; (2) negative-gradient exploration, which systematically uses gradients from failed trajectories to steer the policy away from known failure modes; and (3) a difficulty-budget curriculum that couples task difficulty with the training token budget. A 1.7B model trained with E3 achieves the best results among same-size models on AIME'25 and HMMT'25, improving on both pass@1 and pass@k, and—crucially—continues to improve when the test budget doubles relative to the training budget. The work marks a paradigm shift from optimizing fixed-budget pass@1 toward training models to explore effectively, enabling sustainable gains without scaling model size. Open questions include automatic skill discovery, negative-gradient training stability, and cross-domain generalization.

E3: When Test-Time Compute Learns to Explore—Asymmetric Skill Chaining and Negative-Gradient Reasoning Extrapolation

> In June 2025, a CMU team released E3 (*Learning to Explore Enables Extrapolation of Test-Time Compute*), exposing a structural flaw in existing reasoning models: they fail to extrapolate test-time compute beyond the budget seen during training. By introducing three key ingredients—asymmetric skill chaining, negative-gradient exploration, and a difficulty-budget coupled curriculum—E3-trained 1.7B models reach state-of-the-art results among same-scale models on AIME'25 and HMMT'25, and extrapolate to 2x the training budget.

---

1. The Core Challenge: The Extrapolation Dilemma

1.1 From Interpolation to Extrapolation

Test-time compute research typically focuses on interpolation—optimizing performance within the budget range seen during training. The harder problem is extrapolation:

> Extrapolation: A model optimized with training budget $C_{\text{train}}$ —can it effectively use $C_{\text{test}} \gg C_{\text{train}}$ at test time to keep improving?

Setlur et al. (2025) show that most existing reasoning models fail on this dimension:

> Key finding: Current training objectives typically optimize single-shot pass@1 within a fixed budget, rather than sustained improvement under variable budgets. Models learn to "answer within the budget," not to "keep exploring with more budget."

---

2. E3's Three-Component Framework

2.1 Component 1: Asymmetric Skill Chaining

E3's core insight: base models differ greatly across sub-skills. Rather than improving all abilities uniformly, leverage strong skills to compensate for weak ones via chained composition that performs in-context search.

| Skill | Typical difficulty | Mastery | Role in the chain | |:---|:---:|:---:|:---| | Verification | Low | Strong | Filter: evaluate candidates | | Generation | Medium | Moderate | Producer: generate candidates | | Refinement | High | Weak | Improver: refine with feedback |

> Chaining mechanism: The model doesn't need expert-level generation. It only needs to produce sufficiently diverse candidates, then let its strong verification skill filter them. This "play to strengths" strategy lets overall reasoning exceed any single skill.

Formally, with skills $\mathcal{S} = \{s_1, s_2, \dots, s_n\}$ and mastery levels $c_i \in [0, 1]$ , E3 seeks a skill sequence $\sigma = (s_{i_1}, s_{i_2}, \dots, s_{i_k})$ maximizing chained success:

P(\text{success} \mid \sigma) = \prod_{j=1}^{k} P(s_{i_j} \text{ succeeds} \mid \text{history}_{j-1})

2.2 Component 2: Negative-Gradient Exploration

Standard RL mainly uses positive signals from correct trajectories. E3's innovation: systematically use negative gradients from failed trajectories to guide exploration.

For a failed trajectory $\tau^-$ , the conventional update is:

\nabla_\theta J \approx 0 \quad (\text{since } r(\tau^-) = 0)

E3 converts it into an exploration signal:

\nabla_\theta J_{\text{explore}} \propto -\nabla_\theta \log \pi_\theta(\tau^-)

> Mechanism: The negative gradient indicates how to decrease the probability of the failed trajectory—equivalently increasing the probability of exploring others. Amplifying this signal pushes the model away from known failure modes into under-explored reasoning space.

2.3 Component 3: Difficulty-Budget Curriculum

E3 explicitly couples task difficulty with the training token budget:

| Training phase | Difficulty mix | Budget | Learning goal | |:---:|:---:|:---:|:---| | Early | Mostly easy | Short | Master basic skill chains | | Middle | Mixed | Medium | Extend chain length | | Late | Mostly hard | Long | Complex multi-step exploration |

> Rationale: Easy problems don't need long exploration chains—forcing long budgets causes overthinking. Hard problems need room to explore; short budgets cap the chance of finding solutions. Matching difficulty to budget teaches the model to allocate compute on demand.

---

3. Results: SOTA Performance from a 1.7B Model

3.1 Benchmark Performance

| Benchmark | E3-1.7B | Same-scale baselines | Key advantage | |:---:|:---:|:---:|:---| | AIME'25 | Best 1.7B model | Clearly behind | Strong at both pass@1 and pass@k | | HMMT'25 | Best 1.7B model | Clearly behind | Diverse candidate generation |

> Dual advantage: E3 improves both single-shot pass@1 and pass@k (at-least-one-correct among multiple candidates), indicating the model learned to generate diverse reasoning paths rather than over-optimizing one strategy.

3.2 Extrapolation Verification

> Core evidence: Trained only with a 1K-token budget, E3 models continue to improve under a 2K-token test budget—verifying that the model learned an exploration strategy itself, not behavior tied to a specific budget.

---

4. Theoretical Context: E3 vs. Related Frameworks

4.1 Relation to MRT

E3 complements CMU's earlier MRT (Meta Reinforcement Fine-Tuning) work:

| Dimension | MRT | E3 | |:---|:---|:---| | Core goal | Minimize cumulative regret | Maximize extrapolation | | Mechanism | Dense progress reward | Skill chaining + negative gradients | | Granularity | Episode-level | Skill-level | | Insight | Every episode must make progress | Model must learn to explore |

> Potential synergy: E3's exploration and MRT's progress reward may stack—optimizing per-step progress during exploration could yield stronger test-time compute gains.

4.2 Contrast with Standard RL

| Feature | Standard outcome-reward RL | E3 | |:---|:---|:---| | Signal source | Correct trajectories only | Correct + failed trajectories | | Exploration | Random sampling | Negative-gradient-guided, systematic | | Skill assumption | Uniform capability | Asymmetric multi-skill composition | | Budget handling | Fixed | Difficulty-adaptive |

---

5. Limitations and Open Questions

5.1 Automating Skill Chaining

E3 currently requires manually defined skill sets and chains. Open problems:

Can skill decompositions be discovered automatically from data?
Is the optimal chain structure task-dependent?
Is dynamic chain adjustment (choosing the next skill from intermediate results) feasible?

5.2 Training Stability of Negative Gradients

Amplifying negative gradients may destabilize training:

How to balance positive vs. negative gradient ratios?
Could negative gradients cause mode collapse into overly conservative policies?
Does the guidance remain effective in more complex reward landscapes?

5.3 Cross-Domain Generalization

E3 has mainly been validated on mathematical reasoning:

Code generation: verification (compiling/tests) strong, generation weak—does the asymmetry hold?
Scientific reasoning: are skill decompositions equally clear?
Multimodal reasoning: is cross-modal verification feasible?

---

6. Conclusion

E3 marks a paradigm shift for test-time compute: from "optimize performance within a fixed budget" to "train models to explore effectively, enabling budget extrapolation." Its three ingredients—asymmetric skill chaining, negative-gradient exploration, and the difficulty-budget curriculum—form a complete training framework that lets a small 1.7B model reach unprecedented performance through smart exploration.

As pretraining costs keep climbing, E3 represents a more sustainable path: not making models bigger, but making models smarter with the compute they already have.

---

Paper Details

| Item | Content | |:---|:---| | Title | E3: Learning to Explore Enables Extrapolation of Test-Time Compute for LLMs | | Authors | Amrith Setlur, Matthew Y.R. Yang, Charlie Snell, Jeremy Greer, Ian Wu, Virginia Smith, Max Simchowitz, Aviral Kumar | | Institution | Carnegie Mellon University | | arXiv ID | 2506.09026 | | Date | 2025-06-10 | | Contributions | Asymmetric skill chaining; negative-gradient exploration; difficulty-budget curriculum; test-time compute extrapolation | | Key result | E3-1.7B is the best 1.7B model on AIME'25 and HMMT'25; extrapolates to 2x the training budget |

E3: When Test-Time Compute Learns to Explore—Asymmetric Skill Chaining and Negative-Gradient Reasoning Extrapolation

Summary

E3: When Test-Time Compute Learns to Explore—Asymmetric Skill Chaining and Negative-Gradient Reasoning Extrapolation

1. The Core Challenge: The Extrapolation Dilemma

1.1 From Interpolation to Extrapolation

2. E3's Three-Component Framework

2.1 Component 1: Asymmetric Skill Chaining

2.2 Component 2: Negative-Gradient Exploration

2.3 Component 3: Difficulty-Budget Curriculum

3. Results: SOTA Performance from a 1.7B Model

3.1 Benchmark Performance

3.2 Extrapolation Verification

4. Theoretical Context: E3 vs. Related Frameworks

4.1 Relation to MRT

4.2 Contrast with Standard RL

5. Limitations and Open Questions

5.1 Automating Skill Chaining

5.2 Training Stability of Negative Gradients

5.3 Cross-Domain Generalization

6. Conclusion

Paper Details

Tags