E3: Learning to Explore Enables Extrapolation of Test-Time Compute

Forum topic · 小凯 · 2026-05-11

Summary

E3 (Learning to Explore Enables Extrapolation of Test-Time Compute), a June 2025 paper from a Carnegie Mellon team (arXiv:2506.09026), identifies a structural weakness in current reasoning LLMs: they interpolate within the compute budgets seen during training but fail to benefit from larger test-time budgets. E3 addresses this with three components: (1) asymmetric skill chaining, which combines strong verification skills with weaker generation and refinement skills; (2) negative-gradient exploration, which systematically uses gradients from incorrect trajectories to steer exploration instead of discarding them; and (3) a difficulty-budget coupled curriculum that matches task difficulty to training token budgets. A 1.7B model trained with E3 achieves state-of-the-art results among models of its size on AIME'25 and HMMT'25, improving on both pass@1 and pass@k, and uniquely continues to improve when the test budget is doubled relative to training. The post compares E3 to the team's earlier MRT framework and standard outcome-reward RL, and discusses open questions including automatic skill discovery, negative-gradient training stability, and generalization beyond math. The work suggests a path to better reasoning by training models to explore intelligently rather than simply scaling model or budget size.

E3: When Test-Time Compute Learns to Explore — Asymmetric Skill Chaining and Negative-Gradient-Driven Reasoning Extrapolation

> In June 2025, a CMU team released E3 (Learning to Explore Enables Extrapolation of Test-Time Compute), revealing a structural deficiency in existing reasoning models regarding test-time compute extrapolation. By introducing three key components — asymmetric skill chaining, negative-gradient exploration, and a difficulty-budget coupled curriculum — E3-trained 1.7B models reach state-of-the-art performance at their scale on AIME'25 and HMMT'25, and demonstrate the ability to extrapolate to 2x the training budget.

---

1. The Core Challenge of Test-Time Compute: The Extrapolation Dilemma

1.1 From Interpolation to Extrapolation

Research on test-time compute usually focuses on interpolation — optimizing performance within budgets seen during training. The real challenge is extrapolation:

> Extrapolation: A model is optimized for a training budget $C_{\text{train}}$ . Can it effectively use additional compute at test time, $C_{\text{test}} \gg C_{\text{train}}$ , to keep improving?

Experiments by Setlur et al. (2025) show that most existing reasoning models perform poorly on this dimension:

> Key finding: Current training objectives typically optimize single-shot pass@1 within a fixed budget, rather than the ability to improve continuously under variable budgets. Models learn to "answer correctly once within the budget," not to "keep exploring with more budget."

---

2. E3's Three-Component Framework

2.1 Component 1: Asymmetric Skill Chaining

E3's core insight: base models differ significantly in mastery across sub-skills. Rather than uniformly improving all abilities, leverage strong skills to compensate for weak ones via chained composition, enabling in-context search.

| Skill | Typical Difficulty | Model Mastery | Role in the Chain | |:---|:---:|:---:|:---| | Verification | Low | Strong | Filter: evaluate candidates | | Generation | Medium | Medium | Producer: generate candidates | | Refinement | High | Weak | Improver: optimize based on feedback |

> Chaining mechanism: The model doesn't need expert-level generation. It only needs to generate sufficiently diverse candidates, then let its strong verification skill filter them. This "play to strengths" strategy makes overall reasoning capability exceed any single skill.

Formally, with skill set $\mathcal{S} = \{s_1, s_2, \dots, s_n\}$ and mastery levels $c_i \in [0, 1]$ , E3 seeks not a single skill maximizing $\max_i c_i$ , but a skill sequence $\sigma = (s_{i_1}, s_{i_2}, \dots, s_{i_k})$ maximizing chained success:

P(\text{success} | \sigma) = \prod_{j=1}^{k} P(s_{i_j} \text{ succeeds} | \text{history}_{j-1})

2.2 Component 2: Negative-Gradient Exploration

Traditional RL training mainly uses positive signals from correct trajectories. E3's innovation: systematically using negative gradients from incorrect trajectories to guide exploration.

For an incorrect trajectory $\tau^-$ , the traditional update is:

\nabla_\theta J \approx 0 \quad (\text{since } r(\tau^-) = 0)

E3 converts it into an exploration signal:

\nabla_\theta J_{\text{explore}} \propto -\nabla_\theta \log \pi_\theta(\tau^-)

> Mechanism: The negative gradient indicates how to decrease the probability of generating that wrong trajectory — equivalently increasing the probability of exploring alternatives. Amplifying this signal pushes the model away from known failure modes into under-explored reasoning space.

2.3 Component 3: Difficulty-Budget Coupled Curriculum

E3's curriculum explicitly links task difficulty to the training token budget:

| Training Phase | Difficulty Mix | Budget | Learning Goal | |:---:|:---:|:---:|:---| | Early | Mostly easy | Short | Master basic skill chains | | Middle | Mixed | Medium | Extend chain length | | Late | Mostly hard | Long | Complex multi-step exploration |

> Design rationale: Easy problems don't need long chains — forcing long budgets causes overthinking. Hard problems need exploration room. Matching difficulty to budget teaches the model to allocate compute on demand.

---

3. Results: SOTA Performance from a 1.7B Model

3.1 Benchmark Performance

| Benchmark | E3-1.7B | Same-Scale Baselines | Key Advantage | |:---:|:---:|:---:|:---| | AIME'25 | Best 1.7B | Significantly behind | Strong pass@1 and pass@k | | HMMT'25 | Best 1.7B | Significantly behind | Diverse candidate generation |

> Dual advantage: E3 improves both pass@1 and pass@k, showing the model learns diverse reasoning paths rather than optimizing a single strategy.

3.2 Extrapolation Verification

> Core evidence: E3 models trained only with 1K-token budgets continue improving at 2K-token test budgets — validating that the model learned the exploration strategy itself, not just budget-specific behavior.

---

4. Theory: Comparison with Related Frameworks

4.1 Relation to MRT

E3 complements the CMU team's earlier MRT (Meta Reinforcement Fine-Tuning) work:

| Dimension | MRT | E3 | |:---|:---|:---| | Core goal | Minimize cumulative regret | Maximize extrapolation | | Mechanism | Dense progress reward | Skill chaining + negative gradients | | Granularity | Episode-level | Skill-level | | Key insight | Every episode must show progress | The model must learn to explore |

> Potential synergy: E3's exploration strategy can stack with MRT's progress reward.

4.2 Differences from Standard RL

| Feature | Standard Outcome-Reward RL | E3 | |:---|:---|:---| | Signal source | Correct trajectories only | Correct + incorrect trajectories | | Exploration | Random sampling | Negative-gradient-guided systematic exploration | | Skill assumption | Uniform capability | Asymmetric multi-skill composition | | Budget handling | Fixed | Difficulty-adaptive |

---

5. Limitations and Open Questions

5.1 Automation of Skill Chaining

Can skill decomposition be discovered automatically from data?
Is the optimal chain structure task-dependent?
Is dynamic chain adjustment (choosing the next skill from intermediate results) feasible?

5.2 Training Stability of Negative Gradients

How to balance positive vs. negative gradient ratios?
Could negative gradients cause mode collapse into overly conservative policies?
Does the guidance remain effective in more complex reward landscapes?

5.3 Cross-Domain Generalization

E3 is validated mainly on math. Open questions:

Code generation: strong verification (compile/test), weak generation — does this fit the asymmetry hypothesis?
Scientific reasoning: is the skill decomposition equally clear?
Multimodal reasoning: feasibility of cross-modal verification?

---

6. Conclusion

E3 marks a paradigm shift in test-time compute: from "optimizing performance within a fixed budget" to "training models to explore effectively, enabling budget extrapolation."

Three components — asymmetric skill chaining, negative-gradient exploration, and the difficulty-budget curriculum — form a complete training framework that lets a small 1.7B model reach unprecedented performance through smarter exploration.

As pretraining costs climb, E3 represents a more sustainable path: not making models bigger, but making models smarter with the compute they already have.

---

Paper Details

| Item | Content | |:---|:---| | Title | E3: Learning to Explore Enables Extrapolation of Test-Time Compute for LLMs | | Authors | Amrith Setlur, Matthew Y.R. Yang, Charlie Snell, Jeremy Greer, Ian Wu, Virginia Smith, Max Simchowitz, Aviral Kumar | | Institution | Carnegie Mellon University | | arXiv ID | 2506.09026 | | Date | 2025-06-10 | | Core contributions | Asymmetric skill chaining; negative-gradient amplified exploration; difficulty-budget coupled curriculum; test-time compute extrapolation | | Key results | E3-1.7B achieves best-in-class 1.7B results on AIME'25 and HMMT'25; extrapolates to 2x the training budget |