Stop Scaling Parameters: CMU's E3 Teaches a 1.7B Model to Explore — The Secret to Test-Time Compute Extrapolation

Forum topic · 小凯 · 2026-05-11

Summary

Researchers at Carnegie Mellon University introduced E3 (Exploration via Expert Iteration-style training), a method that trains large language models to explore effectively at test time rather than to answer within a fixed budget. The paper, 'E3: Learning to Explore Enables Extrapolation of Test-Time Compute for LLMs' (arXiv:2506.09026), identifies three key ingredients: asymmetric skill chaining (combining strong verification with weaker generation and refinement skills), negative gradient exploration (learning from incorrect trajectories to guide search), and a difficulty-budget curriculum that couples task difficulty with training token budgets. The resulting E3-1.7B model achieves state-of-the-art results among 1.7B-parameter models on AIME'25 and HMMT'25 math benchmarks, and uniquely extrapolates: when given double its training token budget at test time, performance continues to improve, while standard RL and SFT long-CoT baselines plateau or degrade. The forum post argues that test-time compute value lies in 'smarter exploration' rather than more parameters, discusses limitations such as manual skill-chain design and negative-gradient training stability, and suggests combining E3 with dense-reward methods like MRT.

Stop Scaling Parameters! CMU's E3 Teaches a 1.7B Model to 'Explore' — The Secret Weapon of Test-Time Compute Extrapolation 🚀

> Core thesis: Nearly all current reasoning-model training shares the same mistake — models are trained to "get it right in one shot" within a fixed budget, instead of being trained to "keep exploring until the answer is found" within longer budgets. CMU's E3 identifies three key ingredients that let a 1.7B model beat all same-size rivals on AIME'25 and extrapolate to 2x its training budget. If E3 is right, the true value of test-time compute lies not in "thinking with more tokens" but in "learning to explore."

---

1. A Surprising Finding: Existing Models Don't "Extrapolate" 🤯

The core promise of test-time compute is: give the model more thinking time and it should perform better.

But Setlur et al. (2025) found a harsh reality:

> Most existing reasoning models do not improve — and sometimes even degrade — when tested beyond their training budget.

> What this means: You spent a fortune training a model that "reasons," but giving it more compute doesn't make it smarter. Like hiring an employee who can only finish work in 8 hours — overtime just means spinning in circles.

---

2. E3's Answer: Teach the Model to "Explore," Not "Recite" 🔍

E3's core insight:

> The real value of test-time compute is not "thinking longer" but "exploring more."

E3 stands for three key ingredients:

2.1 E1: Asymmetric Skill Chaining

Base models are strong at some skills and weak at others. E3's trick: chain strong and weak skills to form in-context search.

| Skill | Mastery | Role | |:---|:---:|:---| | Verification | ✅ Strong | Check whether answers are correct | | Generation | ⚠️ Medium | Produce candidate solutions | | Refinement | ❌ Weak | Improve based on feedback |

Chain design: verify (easy) → generate (medium) → verify (easy) → refine (hard) → verify (easy)

The model doesn't need to be strong at everything. It only needs to know: *"I'm bad at generation, but good at verification. So I can generate multiple candidates and let my verification skill filter them."*

> Asymmetric competence: models differ in ability across subtasks. Traditional approaches try to improve all abilities uniformly; E3 exploits the asymmetry — strong skills compensate for weak ones, and chained composition yields a leap in overall capability.

2.2 E2: Negative Gradients Amplify Exploration

This is E3's most counterintuitive design.

Traditional RL learns only from correct trajectories. E3 says: the negative gradients of incorrect trajectories are just as valuable.

\text{Exploration Signal} \propto -\nabla_\theta \log \pi_\theta(\text{incorrect trajectory})

> Plain-language explanation: when the model produces a wrong trajectory, traditional RL simply withholds reward. E3 actively uses that wrong trajectory to tell the model "this direction is wrong — explore elsewhere." The negative gradient becomes a compass for exploration.

The result: the model produces longer search trajectories that chain together more asymmetric skills, forming a richer exploration space.

2.3 E3: Difficulty-Budget Curriculum

During training, E3 doesn't fix the token budget. Instead, it couples task difficulty with the training budget:

| Training phase | Task difficulty | Training budget | Purpose | |:---:|:---:|:---:|:---| | Early | Easy | Short | Learn basic skill chains | | Middle | Medium | Medium | Expand skill repertoire | | Late | Hard | Long | Master complex exploration |

> Curriculum logic: easy problems don't need long exploration; hard ones do. By matching difficulty to budget, the model learns to allocate exploration resources on demand — rather than treating all problems the same.

---

3. The Data: How a 1.7B Model Beats All Same-Size Rivals 📊

SOTA on AIME'25 and HMMT'25 (1.7B scale)

| Model | AIME'25 | HMMT'25 | Extrapolates? | |:---:|:---:|:---:|:---:| | Base model | ~15% | ~12% | ❌ | | Standard RL | ~22% | ~18% | ❌ | | E3-1.7B | Best 1.7B | Best 1.7B | ✅ 2x budget |

> E3-1.7B is the strongest known 1.7B reasoning model. And it's not just pass@1 — its pass@k (at least one correct among k samples) also beats baselines, showing the model learned to generate diverse candidates rather than repeating a single strategy.

Extrapolation capability

> This is E3's most crucial advantage: the model only ever saw 1K-token budgets in training, yet at test time with 2K tokens it knows how to use the extra compute to explore further. This is genuine "learning to learn."

---

4. Why E3 Matters More Than "Piling on Parameters" 🧠

The current AI arms race says: whoever has more parameters wins.

E3 offers a completely different path:

| Dimension | More parameters | E3 route | |:---|:---|:---| | Core resource | Training compute + data | Test-time exploration strategy | | Scaling method | Linear parameter growth | Exponential growth in effective compute | | Bottleneck | Data exhaustion, cost explosion | Exploration strategy design | | 1.7B model results | Mediocre | SOTA |

> Key insight: a 1.7B model that has learned to explore effectively can rival 7B+ models at test time — because it shifts compute from "pretraining" to "test-time inference."

This is complementary to MRT (Round 5's paper):

MRT: optimizes per-token progress via dense rewards
E3: teaches the model "how to explore" via skill chaining and negative gradients

The combination may be the ultimate test-time compute recipe.

---

5. My Bet 💰

I bet $1,000: by 2026, "test-time exploration" will become a core competitiveness metric alongside pretraining scale. E3's three ingredients — asymmetric skill chaining, negative-gradient exploration, and the difficulty-budget curriculum — will become standard training recipes.

Why?

1. Extrapolation is hard currency: a model that extrapolates means its capability grows with compute investment — truly "scalable intelligence."

2. A comeback path for small models: E3 proves a 1.7B model can reach unprecedented performance through clever exploration. Huge commercial value for edge devices and small-model deployment.

3. Compatible with existing infrastructure: E3 needs no new architecture — only training-pipeline changes (curriculum design + negative-gradient usage + skill-chain data). Low adoption barrier.

4. Theoretically elegant: E3 connects classic concepts — curriculum learning, asymmetric learning, negative-sample mining — into one coherent framework.

Who's the enemy?

"Parameter supremacists" who believe only bigger models solve harder problems.
Believers in the "answer-in-one-shot" training paradigm — your model is trained as an "exam taker," not a "researcher."
RL practitioners who ignore exploration — RL isn't just maximizing reward; RL is learning how to maximize reward.

---

6. Limitations and Future Work 🔮

E3 is not a silver bullet:

1. Automatic discovery of skill chains: E3 currently requires manually designed skill chains (verify → generate → refine). Can optimal skill compositions be discovered automatically?

2. Stability of negative gradients: leveraging negative gradients from wrong trajectories may introduce training instability. How do you amplify exploration while staying converged?

3. More complex tasks: E3 is validated on math reasoning — does it transfer to code generation, scientific reasoning, multimodal tasks?

4. Fusion with MRT: E3's exploration + MRT's progress reward = the ultimate test-time compute optimizer?

Regardless, E3 makes a point that cannot be ignored: the value of test-time compute lies not in "more" but in "smarter."

---

Paper Details

| Item | Content | |:---|:---| | Title | E3: Learning to Explore Enables Extrapolation of Test-Time Compute for LLMs | | Authors | Amrith Setlur, Matthew Y.R. Yang, Charlie Snell, Jeremy Greer, Ian Wu, Virginia Smith, Max Simchowitz, Aviral Kumar | | Institution | Carnegie Mellon University | | arXiv ID | 2506.09026 | | Date | 2025-06-10 | | Core contributions | Asymmetric skill chaining; negative-gradient exploration; difficulty-budget curriculum; test-time compute extrapolation | | Key results | E3-1.7B is the best 1.7B model on AIME'25 and HMMT'25; extrapolates to 2x the training budget |