Fixed-Point Reasoners: Teaching AI When to Stop Thinking with a Single Mathematical Formula

Forum topic · 小凯 · 2026-06-17

Summary

A deep-dive into the paper 'Fixed-Point Reasoners: Stable and Adaptive Deep Looped Transformers' (Movahedi et al., ETH Zurich, CSCS, University of Zurich). The proposed FPRM architecture replaces fixed-depth Transformers with a looped Transformer that stops iterating when its hidden state converges to a fixed point—no separate halting module required. Key innovations include Pre-Norm with learnable residual scaling, which the authors prove yields bounded iterates (Theorem 1) and convergence to a unique fixed point via contraction mapping (Theorem 2), plus a damped-update scheme to suppress oscillations (Theorem 3). Training uses truncated BPTT with provably exponentially decaying gradient error. With only 7M parameters, FPRM achieves 94.2% on Sudoku-Extreme (surpassing the specialized EqR), 87.0% on Maze-Hard, 98.1%/98.8% on A5/S5 state tracking, and 47.5% on ARC-1, outperforming TRM baselines while using ~27% fewer effective layers. The authors also propose that hierarchical recursions in prior models (TRM, HRM) may merely compensate for post-norm signal propagation problems rather than mimicking human cognition.

> *A deep-dive into Fixed-Point Reasoners* > > *Paper: Movahedi et al., "Fixed-Point Reasoners: Stable and Adaptive Deep Looped Transformers", arXiv:2606.18206*

---

The Core Idea: Convergence as a Stop Signal

How does a reasoning system know when it has thought enough? The authors answer with fixed-point iteration: a looped Transformer updates its hidden state

z_{i+1} = f_\theta(z_i; x)

and halts when $z_{i+1} \approx z_i$ . When the state stops changing, the answer is ready. No external halting module, no discrete stop decisions, no threshold tuning—convergence itself is the signal.

Traditional Transformers are "depth-fixed": the same number of layers regardless of whether the question is "1+1" or a hard proof. FPRM lets the model adapt its thinking depth to problem difficulty.

Why Looped Models Previously Failed

1. Signal propagation: Post-Norm stabilizes training but blocks information flow, so performance saturates early and depth is wasted. 2. Convergence detection: Prior methods like ACT (Adaptive Computation Time) require a separately trained halting module with difficult discrete-to-continuous relaxations, and can be unstable across seeds.

The Solution: Pre-Norm + Residual Scaling

FPRM combines Pre-Norm with learnable residual scaling factors:

Intra-layer scaling (Eq. 2):

z_\ell = \alpha_1 z_{\ell-1} + \beta_1 f_\ell^{\theta_\ell}(\text{Norm}_{pre}(z_{\ell-1}))

Inter-iteration scaling (Eq. 3):

z_{i+1}^0 = \alpha_2 z_i^{2L} + \beta_2 x

| Property | Post-Norm | Plain Pre-Norm | Pre-Norm + Residual Scaling (FPRM) | |:---|:---|:---|:---| | Training stability | ✓ | ✗ diverges | ✓ stable | | Signal propagation | ✗ poor | ✓ good | ✓ good, depth is used | | Convergence guarantee | none | none | ✓ mathematically proven |

Two theorems back this up:

Theorem 1 (Boundedness): With $0 \leq \alpha_1, \alpha_2 < 1$ and appropriate $\beta_1, \beta_2$ , the iterate norms stay bounded—no explosion.
Theorem 2 (Convergence): When $\alpha_2 \lambda_f < 1$ (with Lipschitz constant $\lambda_f$ ), the map is a contraction, guaranteeing linear convergence to a unique fixed point (Banach's fixed-point theorem, 1922).

Damped Updates Against Oscillation

If oscillation is detected (residual not decreasing), a damping factor blends old and new states:

z \leftarrow \eta \cdot \hat{z} + (1-\eta) \cdot z

Theorem 3 shows damping ensures convergence even when the Jacobian has eigenvalues causing oscillation, without shifting the fixed point.

Training: Truncated BPTT

Backpropagating through hundreds of loop steps would exhaust GPU memory. FPRM uses truncated BPTT:

\frac{dz^\star}{d\theta} = (I-J)^{-1}P \approx \sum_{j=0}^{k-1} J^j P

with a proven bound on truncation error,

\|(I-J)^{-1} - \sum_{j=0}^{k-1}J^j\|_F \leq \frac{\sqrt{D}\sigma^k}{1-\sigma}

i.e., the error decays exponentially—k=4 or k=8 suffices in practice.

Experiments: 7M Parameters, Big Results

| Task | Model | Result | |:---|:---|:---| | Sudoku-Extreme | TRM baseline (7M) | 74.7% | | Sudoku-Extreme | EqR (7M, Sudoku-specialized) | 93.0% | | Sudoku-Extreme | FPRM (7M) | 94.2% | | Maze-Hard | FPRM (7M) | 87.0% | | State tracking A5 (len 128) | TRM / TRM+ACT / FPRM | 45.8% / 65.3% / 98.1% | | State tracking S5 (len 128) | TRM / TRM+ACT / FPRM | 39.4% / 96.2% / 98.8% | | ARC-1 / ARC-2 | FPRM (7M) | 47.5% / 6.2% |

Notable findings:

FPRM reaches equal accuracy using ~27% fewer effective layers than TRM—easy puzzles get fewer loops, hard ones more.
Unlike TRM+ACT, FPRM shows smooth, reliable adaptive computation with no seed failures.
Residual scaling parameters evolve during training: initializing $\alpha_1=0.75, \alpha_2=0.25$ gives 94.23% on Sudoku vs. 83.24% for the reverse. After training, $\alpha_2$ concentrates at smaller values—the model learns to contract itself.

A Paradigm-Flipping Insight on Hierarchy

Prior looped models (TRM, HRM) use hierarchical recursion, often justified as mimicking Kahneman's System 1/System 2 cognition. The authors propose a counter-hypothesis:

> "The success of hierarchy may not stem from cognitive motivations, but as an indirect workaround for post-norm signal propagation problems."

FPRM solves signal propagation directly via architecture (Pre-Norm + residual scaling) and beats hierarchical models without any hierarchy—an Occam's razor moment suggesting future looped architectures should fix the root cause rather than engineer around symptoms.

Takeaway

FPRM formalizes thinking as iterative refinement until the mental state converges: rough idea → repeated transformation → convergence ("I've got it") → answer. The model shifts from a passive fixed-depth executor to an active thinker that adapts reasoning depth to problem difficulty—a small formula ( $$x^* = f(x^*)$$ ) teaching machines when to stop.

References

1. Movahedi, S., et al. (2026). *Fixed-Point Reasoners: Stable and Adaptive Deep Looped Transformers*. arXiv:2606.18206. 2. Banach, S. (1922). *Sur les opérations dans les ensembles abstraits et leur application aux équations intégrales*. Fundamenta Mathematicae. 3. Dehghani, M., et al. (2019). *The Universal Transformer*. ICLR 2019. 4. Graves, A. (2016). *Adaptive Computation Time for Recurrent Neural Networks*. arXiv:1603.08983. 5. Chollet, F. (2019). *On the Measure of Intelligence*. arXiv:1911.01547.