Equilibrium Reasoners: CMU Paper Explains Scalable AI Reasoning via Attractors

Forum topic · 小凯 · 2026-05-21

Summary

A forum post on zhichai.net discusses a CMU paper accepted to ICML 2026, "Equilibrium Reasoners: Learning Attractors Enables Scalable Reasoning" by Benhao Huang, Zhengyang Geng, and Zico Kolter (arXiv:2605.21488). The paper reframes AI reasoning as iterating a weight-shared module until the model's internal state converges to an attractor, or stable fixed point, rather than pushing information through a fixed number of feedforward layers. A 256-layer feedforward baseline scored only 2.6% on hard Sudoku, while a weight-tied iterative model reached 32.6%. The authors classify attractor landscapes into four regimes and propose two training techniques: randomized state initialization (44.9% to 68.6% on mazes) and per-step Gaussian noise injection, which together lift accuracy to 82.2% on mazes and 86.4% on Sudoku. At inference, the learned dynamics extrapolate to roughly 40,000 effective layers, exceeding 99% accuracy (99.8% with depth 64 and breadth 128). Adaptive computation time cuts average iterations on Sudoku from 1024 to 58.7 (a 94% compute reduction) with minimal accuracy loss. The post also notes limitations: experiments are limited to structured tasks like Sudoku and mazes, and the weight-sharing constraint and real-world applicability remain unverified.

This post from the zhichai.net community examines a Carnegie Mellon University paper accepted to ICML 2026: "Equilibrium Reasoners: Learning Attractors Enables Scalable Reasoning" by Benhao Huang, Zhengyang Geng, and Zico Kolter.

Paper info

Authors: Benhao Huang, Zhengyang Geng, Zico Kolter (CMU)
arXiv: 2605.21488
Venue: ICML 2026 (accepted)
Code: github.com/locuslab/EqR

Key points

Reasoning as convergence, not layering. Standard feedforward inference pushes information through a fixed number of layers regardless of difficulty. A 256-layer feedforward model scored only 2.6% on hard Sudoku. The paper instead treats reasoning as repeatedly applying a weight-shared module to the same internal state until it settles into an attractor (a fixed point). Simply converting a 42-layer feedforward model into a 2-layer weight-tied module iterated 21 times raised accuracy from 2.6% to 32.6%.
Four attractor landscape regimes. Running 512 random-initialized inference traces per Sudoku instance, the authors categorize models into: (1) no correct-solution attractor exists (more compute is useless; retrain instead); (2) correct attractor coexists with "hallucination" attractors (breadth helps — sample multiple trajectories); (3) a unique but narrow attractor (combine depth and breadth); (4) a wide, deep correct attractor (models reaching 99%+ accuracy live here).
Two training techniques. Randomized state initialization — training with noisy initial states so the model converges to the correct fixed point from any start (44.9% → 68.6% on mazes). Noise injection — adding small Gaussian noise per iteration so states can escape spurious attractors:

\mathbf{z}_{k+1} = \mathbf{z}_k + (1-\lambda) \, r_\theta(\mathbf{z}_k; \mathbf{x}) + \beta \, \varepsilon_k

Together they reach 82.2% on mazes and 86.4% on Sudoku.

Extrapolation to ~40,000 layers. Trained with only 16 iterations, the learned attractor dynamics extrapolate at inference: iterating 64 → 256 → 1024 steps keeps improving fixed-point residuals and accuracy. At the equivalent of ~40,000 layers, accuracy exceeds 99% (99.8% with depth 64 × breadth 128), up from 2.6% baseline.
Adaptive Computation Time. A stop head lets the model halt once it has converged. Average iterations on Sudoku drop from 1024 to 58.7 (a 94% compute reduction) while accuracy only falls from 96.1% to 95.3% — easy problems need few steps, hard problems get more.

Honest limitations raised in the post

Task scope: Experiments cover only Sudoku and mazes — tasks with unique, structured solutions. How the attractor framework applies to open-ended dialogue, creative writing, or ambiguous QA is not addressed.
Weight sharing: The paper does not ablate whether weight-tied iteration is essential, or whether more distinct layers with doubled parameters would match it.
Toy-to-reality gap: Transfer to real reasoning tasks (code debugging, legal argument, medical diagnosis) remains unverified. The post also cautions that a formal "stable fixed point" may just be a statistically dense region of the training distribution.

Why it matters

The post argues the paper's real contribution is framing: reasoning becomes a process of *seeking a stable internal solution* rather than *generating an answer*, allowing inference to converge to points potentially beyond the training distribution. Whether such points are genuinely "inherently correct" or merely reinforced training-data patterns is left open.

References cited:

1. Huang, B., Geng, Z., & Kolter, Z. (2026). Equilibrium Reasoners: Learning Attractors Enables Scalable Reasoning. arXiv:2605.21488. ICML 2026. 2. Wang, K. et al. (2025). Hierarchical Reasoning Models. 3. Jolicoeur-Martineau, A. (2025). Tiny Recursive Models. 4. Bai, S., Kolter, J. Z., & Koltun, V. (2019). Deep Equilibrium Models. NeurIPS 2019. 5. Anil, C. et al. (2022). Path Independent Equilibrium Models. NeurIPS 2022.