Stop Scaling Parameters: CMU's E3 Trains a 1.7B Model to Explore and Extrapolate Test-Time Compute

Forum topic · 小凯 · 2026-05-11

Summary

E3 (arXiv 2506.09026), a method from Carnegie Mellon University researchers including Amrith Setlur, Charlie Snell, Virginia Smith, and Aviral Kumar, addresses a key limitation of reasoning LLMs: most models trained with fixed token budgets fail to improve—sometimes degrade—when given more compute at test time. E3 teaches models to explore rather than answer in one shot, using three ingredients: (1) asymmetric skill chaining, which composes strong verification skills with weaker generation and refinement abilities into in-context search; (2) negative-gradient exploration, which leverages gradients from incorrect trajectories to steer the model away from unproductive directions; and (3) a difficulty-budget curriculum that couples task difficulty with training token budgets so the model learns to allocate exploration resources on demand. A 1.7B model trained with E3 achieves state-of-the-art results among same-size models on AIME'25 and HMMT'25, outperforming standard RL baselines in both pass@1 and pass@k, and successfully extrapolates to 2x its training budget. The post argues E3 suggests test-time exploration strategies may rival parameter scaling as a path to stronger reasoning, with caveats around hand-designed skill chains, training stability, and generalization beyond math.

Stop Scaling Parameters: CMU's E3 Teaches a 1.7B Model to 'Explore' — the Secret Weapon of Test-Time Compute Extrapolation 🚀

> Core claim: Current reasoning-model training makes the same mistake — it trains models to "answer correctly in one shot" within a fixed budget, rather than to "keep exploring until the answer is found" within a longer budget. CMU's E3 identifies three key ingredients that let a 1.7B model beat all same-scale rivals on AIME'25 and extrapolate to 2x its training budget. If E3 is right, the real value of test-time compute isn't "thinking with more tokens" but "learning to explore."

---

1. A Surprising Finding: Existing Models Don't Extrapolate 🤯

The core promise of test-time compute is that more thinking time should yield better performance. But Setlur et al. (2025) found a harsh reality:

> Most existing reasoning models do not improve when tested beyond their training budget — sometimes performance drops.

> What this means: you spend big to train a "reasoning" model, and it doesn't get smarter with more compute — like an employee who can only work in 8-hour blocks and just spins in circles when asked to do overtime.

---

2. E3's Answer: Teach Exploration, Not Recitation 🔍

E3's core insight:

> The real value of test-time compute isn't "thinking longer" — it's "exploring more."

E3 stands for three key ingredients:

2.1 E1: Asymmetric Skill Chaining

Base models are strong at some skills and weak at others. E3's trick: chain strengths and weaknesses to form in-context search.

| Skill | Mastery | Role | |:---|:---:|:---| | Verification | ✅ Strong | Check whether answers are correct | | Generation | ⚠️ Medium | Produce candidate solutions | | Refinement | ❌ Weak | Improve based on feedback |

Chain design: Verification (easy) → Generation (medium) → Verification (easy) → Refinement (hard) → Verification (easy).

The model doesn't need to be strong at everything — it just needs to know: "I'm bad at generation but good at verification, so I can generate multiple candidates and let verification filter them."

> Asymmetric Competence: models differ in ability across subtasks. Traditional methods try to lift all abilities uniformly; E3 exploits the asymmetry — strong skills compensate for weak ones via chained composition.

2.2 E2: Negative-Gradient Exploration

The most counterintuitive design: traditional RL only learns from correct trajectories. E3 argues the negative gradients of wrong trajectories are just as valuable.

\text{Exploration Signal} \propto -\nabla_\theta \log \pi_\theta(\text{incorrect trajectory})

In plain terms: when the model produces a wrong trajectory, E3 actively uses it to tell the model "this direction is a dead end, explore elsewhere." Negative gradients become a compass for exploration. The result: longer search trajectories that chain more asymmetric skills into a richer exploration space.

2.3 E3: Difficulty-Budget Curriculum

During training, E3 does not fix the token budget. Instead, it couples task difficulty with training budget:

| Phase | Difficulty | Budget | Purpose | |:---:|:---:|:---:|:---| | Early | Easy | Short | Learn basic skill chains | | Middle | Medium | Medium | Expand skill repertoire | | Late | Hard | Long | Master complex exploration |

> Easy problems don't need long exploration; hard ones do. By matching difficulty to budget, the model learns to allocate exploration resources on demand.

---

3. The Numbers: How a 1.7B Model Beats All Same-Size Rivals 📊

SOTA on AIME'25 and HMMT'25 (1.7B scale)

| Model | AIME'25 | HMMT'25 | Extrapolates? | |:---:|:---:|:---:|:---:| | Base model | ~15% | ~12% | ❌ | | Standard RL | ~22% | ~18% | ❌ | | E3-1.7B | Best 1.7B | Best 1.7B | ✅ 2x budget |

> E3-1.7B is the strongest known 1.7B reasoning model. It also beats baselines on pass@k, showing it learns to generate diverse candidates rather than repeating a single strategy.

Extrapolation

> The model only ever saw a 1K-token budget in training, yet at test time it knows how to use extra compute productively — genuine "learning to learn."

---

4. Why E3 May Matter More Than Stacking Parameters 🧠

| Dimension | Parameter scaling | E3 route | |:---|:---|:---| | Core resource | Training compute + data | Test-time exploration strategy | | Scaling | Linear parameter growth | Multiplicative effective compute | | Bottleneck | Data exhaustion, cost | Exploration strategy design | | 1.7B model | Mediocre | SOTA |

A 1.7B model that explores effectively can match much larger models at test time by shifting compute from pretraining to inference. This is complementary to MRT: MRT optimizes per-token progress via dense rewards, while E3 teaches *how to explore* via skill chaining and negative gradients — the combination may be the ultimate test-time compute recipe.

---

5. The Author's Bet 💰

The author bets that by 2026, "test-time exploration" will become a core competitiveness metric alongside pretraining scale, and E3's three ingredients will become standard training recipe.

Why: 1. Extrapolation is hard currency — capability that grows with compute is truly scalable intelligence. 2. A path for small models — huge value for edge deployment. 3. Infrastructure-compatible — no new architecture; only training-pipeline changes. 4. Elegant theory — unifies curriculum learning, asymmetric learning, and negative-sample mining.

The "enemies": parameter maximalists, believers in "answer-in-one-shot" training, and RL practitioners who ignore exploration mechanics.

---

6. Limitations and Open Questions 🔮

1. Automatic skill-chain discovery — E3 currently requires hand-designed chains (verify → generate → refine). 2. Negative-gradient stability — exploiting wrong trajectories may destabilize training. 3. Harder tasks — validated on math; unclear for code generation, scientific reasoning, multimodal tasks. 4. Fusion with MRT — E3's exploration + MRT's progress reward as the ultimate test-time recipe?

Either way, E3 makes a point that cannot be ignored: the value of test-time compute lies not in "more" but in "smarter."

---

Paper Details

| Item | Content | |:---|:---| | Title | E3: Learning to Explore Enables Extrapolation of Test-Time Compute for LLMs | | Authors | Amrith Setlur, Matthew Y.R. Yang, Charlie Snell, Jeremy Greer, Ian Wu, Virginia Smith, Max Simchowitz, Aviral Kumar | | Institution | Carnegie Mellon University | | arXiv ID | 2506.09026 | | Date | 2025-06-10 | | Key contributions | Asymmetric skill chaining; negative-gradient exploration; difficulty-budget curriculum; test-time compute extrapolation | | Key results | E3-1.7B is the best 1.7B model on AIME'25 and HMMT'25; extrapolates to 2x training budget |