Have You Ever Had This Experience?
When facing a hard problem on an exam, your first answer feels off. You don't give up—you start over with a different approach, a different starting point. Second try, third try... the answer becomes more and more certain, until something *clicks*: that's it.
That "click" has a mathematical name: convergence to an attractor.
A new paper from CMU Locus Lab (Zico Kolter's group), Equilibrium Reasoners (EqR), argues that neural networks can have the same experience. The key isn't making the network bigger—it's shaping its internal dynamics into a "good" landscape, where the correct answer is a stable attractor and the reasoning process naturally slides toward the right solution.
The result? A tiny 5M-parameter model soars from 2.6% accuracy on Sudoku-Extreme to 99.8%, with an equivalent depth of over 40,000 layers.
Why Does "Thinking More" Sometimes Help and Sometimes Not?
Test-time compute scaling is one of today's hottest AI directions: spend more compute at inference, and accuracy should rise. But in practice, more thinking often hurts—models fall into circular, noisy reasoning loops.
Why? Because repeated thinking only works if your reasoning process has a correct "center of gravity"—an attractor. Without one, extra iterations are wasted.
Attractors: The Gravity of Thought
Imagine a ball rolling over hilly terrain: it settles into the nearest valley—an attractor. The wider and deeper the valley, the more reliably the ball lands there.
EqR's core insight: the internal state space of an iterative reasoning model is exactly such a terrain. Each iteration is one step of the ball's roll. If the landscape is shaped well, the correct answer is the widest, deepest valley.
The authors classify four landscape regimes:
| Regime | Terrain | Outcome | Fix | |--------|---------|---------|-----| | (a) No correct attractor | Only wrong basins | Always wrong | Retraining needed | | (b) Correct and wrong attractors coexist | Good and bad basins | Sometimes right | Try multiple times (breadth scaling) | | (c) Correct attractor too narrow | Good basin tiny, hard to find | Occasionally right | More tries + more steps | | (d) Ideal landscape | Correct basin wide and deep | Always right | EqR's goal |
Key insight: depth (more steps) and breadth (more restarts) are complementary. Breadth only pays off once depth is sufficient—you must reach the basin before more retries help.
From Feedforward to Iterative: A Five-Step Evolution
1. Weight tying — collapse 42 layers of independent parameters into 2 shared layers reused repeatedly. Parameters drop from 105M to 5M, yet accuracy jumps from 2.6% to 32.6%. Shared weights create iterative dynamics—the same function applied over and over, like reconsidering the same problem. 2. Segmented online training (SOT) — update parameters every few iterations rather than after all of them, like checking answers after every few problems instead of after the whole exam. Accuracy: 74.7%. 3. Hierarchical iteration — introduce fast and slow state variables updated at different rates, echoing the brain's fast/slow thinking split. Accuracy: 76.5%. 4. Adaptive computation (ACT) — let the model decide when it has "thought enough": stop early on easy problems, think longer on hard ones. Accuracy: 84.8%. 5. Attractor landscape shaping (EqR's core innovation) — the paper's killer feature.
Two Lightweight Tricks That Reshape the Thinking Terrain
1. Randomized State Initialization (RI)
Earlier iterative models always start training from the same fixed initial state—like always beginning a problem the same way, learning only one path. EqR instead starts each training run from a random position, forcing the model to reach the correct answer from any starting point and greatly widening the correct attractor's basin of attraction. On Maze alone, this lifts accuracy from 44.9% to 68.6%.
2. Noise Injection (NI)
Small Gaussian noise is added at each iteration—like the occasional mental "drift" that helps you escape local optima. With modest noise (β=0.01) and light damping (λ=0.05), Maze accuracy rises from 68.6% to 82.2%.
Synergy: random initialization broadens *where you start*; path noise broadens *how you get there*. Perfectly complementary.
The Numbers: From 2.6% to 99.8%
| Method | Sudoku Extreme | Maze Unique | |--------|---------------|-------------| | Feedforward (64 layers) | 2.6% | 0.0% | | HRM (prior work) | 55.0% | 0.3% | | TRM (prior work) | 84.8% | 44.9% | | URM (prior work) | 77.6% | 51.4% | | EqR (D=16, B=1) | 86.4% | 82.2% | | EqR (D=64, B=1) | 93.0% | 88.9% | | EqR (D=64, B=128) | 99.8% | 93.0% |
Most striking: trained with only 16 iterations, EqR is deployed at 1024+ iterations (equivalent to 40,000+ layers), with accuracy climbing from 86.4% to 99.8%. The attractor dynamics learned during training safely extrapolate to depths far beyond the training range.
Convergence: The Cheapest Answer Quality Inspector
Once the attractor landscape is well shaped, the residual (distance from the current state to the fixed point) becomes a reliable indicator of answer quality. No external verifier needed—just pick the trajectory that converges best. During breadth scaling, selecting the top-1 converged trajectory beats majority voting in both efficiency and accuracy.
Adaptive computation (ACT) lets easy problems halt after ~5.4 iterations on average. At D=1024, average NFEs drop from 1024 to 58.7—a 17.4x efficiency gain with less than 1% accuracy loss.
Engineering Takeaways
1. Small models can do big reasoning. EqR's core model has only 5M parameters. What matters is whether internal dynamics are correctly shaped—significant for edge deployment. 2. Train–inference extrapolation works. Train 16 steps, infer 1024+, and performance keeps improving—true generalization, not overfitting to training depth. 3. Convergence signals beat voting. A mathematical metric (residual) picks the best answer with less compute than majority voting. 4. The two training tricks are universal and lightweight. Randomized initialization and noise injection require no architecture changes, extra data, or external verifiers. 5. Depth and breadth must be combined in the right order. First push depth until a single trajectory can converge, then add breadth for better attractors.
Reflections
The post closes with a deeper question: what are we actually teaching neural networks?
The traditional view says networks learn input–output mappings. EqR suggests iterative models learn a dynamical system—a force field from which correct answers naturally emerge. This mirrors the energy-landscape idea in physics: protein folding, spin glasses, phase transitions—nature's most elegant computations are "descents" on energy landscapes. EqR essentially recreates that principle inside a network.
The notion of "attractor alignment" hints at a new training paradigm: not teaching the network what the answer is, but teaching it how to construct a force field in which the answer naturally appears. That may be a key step toward truly scalable reasoning.
Caveat: current experiments are limited to structured tasks like Sudoku and mazes. Whether attractor landscapes can be cleanly defined and shaped for open-domain reasoning (math proofs, code generation) remains an open question. But this view—reasoning as the evolution of a dynamical system—opens a new door.
---
Paper | arxiv.org/abs/2605.21488 Code | github.com/locuslab/EqR (upcoming release) Authors | Benhao Huang, Zhengyang Geng, Zico Kolter (CMU Locus Lab) Venue | ICML 2026