The Ultimate Secret of AI Reasoning: Turning Thought into a Snowball Rolling into a Deep Valley — CMU's Equilibrium Reasoners (ICML 2026)

Forum topic · 小凯 · 2026-05-21

Summary

A CMU paper accepted at ICML 2026, 'Equilibrium Reasoners: Learning Attractors Enables Scalable Reasoning' by Benhao Huang, Zhengyang Geng, and Zico Kolter (arXiv:2605.21488), argues that scalable AI reasoning comes from treating inference as convergence toward attractors — stable fixed points in the model's internal state space — rather than pushing information through a fixed number of layers. The authors classify four attractor landscape types (no correct attractor, competing false attractors, narrow basin, wide basin) and propose two training techniques: randomized state initialization and per-step noise injection. Replacing a 42-layer feedforward model with a weight-shared 2-layer module iterated repeatedly lifts Sudoku accuracy from 2.6% to 32.6%, and up to 99.8% when iterated to an effective depth of ~40,000 layers. Adaptive Computation Time cuts average iterations from 1024 to 58.7 (94% compute savings) with minimal accuracy loss. The post also candidly notes open questions: applicability to open-ended tasks, the role of weight sharing, and the gap from puzzle benchmarks to real-world reasoning.

A CMU team's paper accepted at ICML 2026 offers a striking insight: the key to scalable large-model reasoning lies in a concept physicists have used for over a century — attractors.

Paper info

| Item | Content | |------|---------| | Title | Equilibrium Reasoners: Learning Attractors Enables Scalable Reasoning | | Authors | Benhao Huang, Zhengyang Geng, Zico Kolter | | Institution | Carnegie Mellon University (CMU) | | arXiv | 2605.21488 | | Venue | ICML 2026 (accepted) | | Code | github.com/locuslab/EqR | | Core claim | Teach the model to treat reasoning as path-finding on an "attractor landscape" — the deeper the thinking, the more the state settles toward the stable correct answer. Easy problems stabilize in one or two steps; extremely hard ones need tens of thousands of iterations, like a snowball rolling into a valley. |

Key points

1. Bulldozer vs. river — why "deeper" isn't always "better"

Current AI models reason like bulldozers: information is pushed layer by layer through a fixed number of layers, with identical compute whether the problem is trivial or brutal. A 256-layer feedforward model achieves only 2.6% accuracy on hard Sudoku — not because it lacks parameters, but because its long chain of forward passes loses information along the way.

The authors' answer: don't treat the network as a bulldozer, treat it as a river — a weight-shared module repeatedly processing the same internal state, each pass nudging the state toward something more consistent, until it can no longer move: the fixed point, the attractor. Converting a 42-layer feedforward model into a 2-layer weight-shared module iterated 21 times lifts accuracy from 2.6% to 32.6% with the same compute.

2. Four attractor landscapes

Running 512 randomly initialized reasoning traces on 256 Sudoku puzzles, the authors identify four patterns:

No correct attractor exists — all traces end far from the answer. More inference compute is wasted; go back to training.
Correct attractor coexists with "hallucination pits" — breadth (sampling more traces) helps more than depth.
A single correct attractor with a too-narrow basin — combine breadth (lottery-ticket odds) and depth (stabilizing near-miss traces).
A wide, deep correct attractor — most traces flow to the right answer automatically; this is where 99%+ accuracy models live.

3. Two hammers: randomized initialization and noise injection

Randomized state initialization: adding noise to initial states during training forces the model to converge correctly from any starting point. On mazes alone, this lifts accuracy from 44.9% to 68.6%.
Path noise injection: a small Gaussian perturbation at every iteration lets the state escape shallow false basins without escaping the true deep one:

\mathbf{z}_{k+1} = \mathbf{z}_k + (1-\lambda) \, r_\theta(\mathbf{z}_k; \mathbf{x}) + \beta \, \varepsilon_k

Together, these push maze accuracy to 82.2% and Sudoku to 86.4% — with no extra inference compute.

4. 40,000 layers deep — the snowball at full speed

Trained with only 16 iterations, the learned attractor dynamics extrapolate at inference time: as iterations scale to 64, 256, 512, 1024, fixed-point residuals drop and accuracy rises. At an effective unrolled depth of ~40,000 layers, accuracy exceeds 99% — up to 99.8% (EqR + depth 64 + breadth 128) versus 2.6% for the feedforward baseline. The parameters didn't change; the way the model uses its internal state did — from one-pass bulldozing to self-reinforcing convergence.

5. Difficulty-aware computation

Adding an Adaptive Computation Time (ACT) stopping head lets the model halt when it has converged. On Sudoku, average iterations drop from 1024 to 58.7 — a 94% compute reduction — with accuracy only falling from 96.1% to 95.3%. The system behaves like a student who answers easy questions instantly and thinks longer on hard ones.

6. Honest limits

Why only Sudoku and mazes? Both have unique, structured solutions — the natural habitat of attractors. The framework's applicability to open-ended dialogue, creative writing, or ambiguous QA is not discussed.
How essential is weight sharing? The paper doesn't compare against 42 distinct layers with doubled parameters. Weight sharing may itself be the key constraint enabling attractors, but that remains a hypothesis.
From toys to reality? The gap between 9×9 grids and real-world reasoning (debugging, legal argument, diagnosis) remains unbridged. "Provable elegance" and "real-world robustness" are still far apart.

7. Why it matters

The individual tricks aren't revolutionary. What's new is the framing: reasoning is no longer "generating an answer" but "finding a stable solution in internal state space." Generation interpolates near the training distribution; convergence to a fixed point can, in principle, reach points never seen in training. Whether such fixed points are genuinely "intrinsically correct" — or just statistically dense regions of the training data iteratively reinforced — remains open. Good science, as this post notes, leaves you with more questions at the last page than the first.

References

1. Huang, B., Geng, Z., & Kolter, Z. (2026). Equilibrium Reasoners: Learning Attractors Enables Scalable Reasoning. arXiv:2605.21488. ICML 2026. 2. Wang, K. et al. (2025). Hierarchical Reasoning Models. arXiv preprint. 3. Jolicoeur-Martineau, A. (2025). Tiny Recursive Models. arXiv preprint. 4. Bai, S., Kolter, J. Z., & Koltun, V. (2019). Deep Equilibrium Models. NeurIPS 2019. 5. Anil, C. et al. (2022). Path Independent Equilibrium Models. NeurIPS 2022.