Overview
Paper: *Solve the Loop: Attractor Models for Language and Reasoning* Authors: Jacob Fein-Ashley, Paria Rashidinejad (USC) arXiv: 2605.12466 (May 12, 2026) Project page: https://attractor-models.github.io/ Code: https://github.com/jacobfa/Attractor
Attractor Models treat iterative refinement as a fixed-point problem rather than a fixed-depth loop. A backbone Transformer produces an initial guess, and a smaller recurrent *attractor* module iterates that guess to a fixed point using Anderson acceleration, with the original guess re-injected at every step.
Key points
Why standard looped Transformers struggle
- Memory grows linearly with loop depth because each iteration must store intermediate activations for backpropagation; Parcae OOMs at 32–64 loops.
- Training is unstable: gradients explode or vanish over long horizons, requiring spectral-radius constraints and normalization tricks.
- Train/test depth mismatch: changing loop count at inference breaks the model because training and inference depths must match.
- Scaling collapse: TRM jumps from 74.7% / 85.3% at 7M parameters on Sudoku-Extreme / Maze-Hard to 0% at 27M — a stability failure of pure recurrent depth.
- 770M Attractor beats a 1.3B Transformer trained on 2× the data.
- Lambada perplexity drops 46.6% at the 140M scale vs Transformer.
- Core accuracy improves up to 19.7% at 770M.
- Training FLOPs reduced 25–31% vs Parcae.
- The recurrent module can be removed at inference for speed.
- Enables adaptive compute: T=0 for easy inputs, more iterations for hard ones.
- Provides a new path to model compression: train looped, deploy feed-forward.
- How large should the attractor module be? No systematic ablation.
- Anderson acceleration's least-squares steps are unfriendly to hardware parallelism.
- Inference latency is not reported; O(1) memory ≠ fast iteration time.
- Multimodal extension untested.
- Relationship to Chain-of-Thought (discrete-token iteration vs latent-space iteration) unexplored.
- arXiv:2605.12466 — Fein-Ashley & Rashidinejad (2026)
- https://attractor-models.github.io/
- https://github.com/jacobfa/Attractor
- Deep Equilibrium Models — Bai et al. (2019)
- Parcae — Prairie et al. (2026)
- Universal Transformers — Dehghani et al. (2019)
- Looped Transformers — Giannou et al. (2023); Yang et al. (2024)
- TRM / HRM — Jolicoeur-Martineau (2025)
- FineWeb-Edu — Penedo et al. (2024)
- Sudoku-Extreme / Maze-Hard — Wang et al. (2025)
- nanochat — Karpathy (2025)
The attractor formulation
The refinement is written as a fixed-point equation:
$$\tilde{y}_{t+1} = T_{\theta_a}(\tilde{y}_t, \tilde{y}_0)$$
Iteration stops when $\tilde{y}_{t+1} \approx \tilde{y}_t$. This generalizes Deep Equilibrium Models (Bai et al., 2019) by:
1. Operating in the output embedding space, not hidden state — a stronger optimization signal. 2. Warm-starting from the backbone's initial guess, not from a zero vector — fewer iterations to converge. 3. Persistent injection of $\tilde{y}_0$ at every step, anchoring the attractor so it cannot drift to an unrelated basin.
O(1) training memory via implicit differentiation
Naive backpropagation through $T$ iterations costs $O(T)$ memory. Attractor Models apply implicit differentiation:
$$\frac{\partial L}{\partial \theta} = u^\top \frac{\partial T_{\theta_a}(\tilde{y}^\star, \tilde{y}_0)}{\partial \theta}, \quad u = (I - J_{\tilde{y}}^\top)^{-1} v$$
In practice, the authors use the one-step approximation $u \approx v$, requiring only one vector-Jacobian product through the attractor. Training memory is constant regardless of how many iterations the forward pass uses (measured ~4.18 GB for 1–64 loops).
Anderson acceleration and adaptive depth
The fixed-point solver uses Anderson acceleration (a quasi-Newton method exploiting past iterates). Stopping criteria are inference-time hyperparameters:
$$\|A_{\theta_a}(\tilde{y}_t, \tilde{y}_0)\|_2 / \|\tilde{y}_t\|_2 < \varepsilon$$
Tighten $\varepsilon$ for accuracy, loosen for speed — no retraining required.
Language modeling results on FineWeb-Edu
Hard reasoning tasks (single forward pass, ~1000 examples each)
| Method | Params | Sudoku-Extreme | Maze-Hard | |---|---:|---:|---:| | DeepSeek R1 | 671B | 0.0% | 0.0% | | Claude 3.7 | — | 0.0% | 0.0% | | o3-mini-high | — | 0.0% | 0.0% | | Transformer | 27M | 0.0% | 0.0% | | HRM | 27M | 55.0% | 74.5% | | TRM | 7M | 74.7% | 85.3% | | TRM | 27M | 0.0% (collapsed) | 0.0% (collapsed) | | Attractor | 7M | 54.3% | 46.7% | | Attractor | 27M | 91.4% | 93.1% |
Frontier LLMs score 0% because the task requires a single direct prediction, not autoregressive generation. Standard Transformers also fail because they lack any iterative refinement mechanism. Attractor Models scale cleanly from 7M → 27M without the TRM collapse.
Equilibrium Internalization
The most surprising finding: after training, the backbone's initial output embedding $\tilde{y}_0$ is already near the fixed point. At 770M, decoding the backbone output directly (T=0, no attractor) matches or exceeds attractor-refined outputs. The attractor functions as a moving teacher during training — the backbone learns to internalize the refinement — analogous to knowledge distillation that happens automatically. Implications:
Comparison with related work
| Method | Loop depth | Train memory | Inference adaptive | Scalable | Stable | |---|:---:|:---:|:---:|:---:|:---:| | Standard Transformer | 1 | O(1) | ❌ | ✅ | ✅ | | Universal Transformer | fixed | O(T) | ❌ | ⚠️ | ⚠️ | | Looped Transformer | fixed | O(T) | ❌ | ⚠️ | ⚠️ | | Parcae | fixed | O(T) | ❌ | ⚠️ | ✅ | | DEQ | adaptive | O(1) | ✅ | ⚠️ | ⚠️ | | Attractor Models | adaptive | O(1) | ✅ | ✅ | ✅ |
Attractor Models combine DEQ's O(1) memory with Parcae-style stability and add a clean scaling story.