Attractor Models Deep Dive: Recurrent Transformer Meets Fixed Points, Iterating Its Way to Answers

Forum topic · 小凯 · 2026-05-14

Summary

This article analyzes 'Solve the Loop: Attractor Models for Language and Reasoning' (arXiv:2605.12466) by Jacob Fein-Ashley and Paria Rashidinejad of USC. Attractor Models replace fixed recurrent depth with fixed-point iteration: a backbone Transformer produces an initial output embedding guess, then a small attractor module refines it to a fixed point using Anderson acceleration, with persistent re-injection of the initial guess. Training uses implicit differentiation with a one-step approximation, giving O(1) memory regardless of iteration count—while baseline Parcae OOMs at 32+ loops. On FineWeb-Edu, the 770M Attractor beats a 1.3B Transformer on perplexity and downstream accuracy while cutting training FLOPs 25-31% vs Parcae. On hard reasoning tasks (Sudoku-Extreme, Maze-Hard) with ~1000 examples, a 27M Attractor reaches 91.4% and 93.1%, far exceeding TRM (which collapses at 27M) and frontier LLMs like DeepSeek R1 (0%). The most striking finding is 'equilibrium internalization': after training, backbone outputs are already near the fixed point, so the attractor can be nearly removed at inference—enabling a train-recurrent, infer-feedforward paradigm.

Attractor Models Deep Dive: When Recurrent Transformers Meet Fixed Points

> Subject: Solve the Loop: Attractor Models for Language and Reasoning > Authors: Jacob Fein-Ashley, Paria Rashidinejad (USC) > arXiv: 2605.12466 > Released: 2026-05-12 > Project page: https://attractor-models.github.io/ > Code: https://github.com/jacobfa/Attractor

1. The Problem: Why Recurrence Alone Isn't the Answer

Standard Transformers finalize each token after a single feedforward pass, but human understanding involves revisiting and refining. Recurrent architectures—Universal Transformer, Looped Transformer, Recurrent-Depth Transformer—iteratively refine a shared computation block, but hit three structural problems:

1. Training memory explosion: backpropagation stores intermediate states per loop iteration, growing linearly with depth. 2. Training instability: gradients explode or vanish over long recurrences. 3. Fixed-depth curse: train at 8 loops, infer at 16—the model breaks (train-test mismatch).

Ironically, tiny recursive reasoners like TRM show a "less is more" pathology: going from 7M to 27M parameters causes performance to collapse to 0%.

2. The Core Idea: Iterate to a Fixed Point

Instead of a preset loop count, Attractor Models treat refinement as a fixed-point problem:

\tilde{y}_{t+1} = T_{\theta_a}(\tilde{y}_t, \tilde{y}_0)

The fixed point ( $\tilde{y}_{t+1} \approx \tilde{y}_t$ ) is the final refined output.

Architecture: Backbone + Attractor

Backbone: a high-capacity causal Transformer producing a "reasonable but imprecise" initial guess.
Attractor: a smaller recurrent network that iterates the guess to the fixed point.
Persistent injection: the initial guess $\tilde{y}_0$ is re-injected at every iteration, preventing drift.

Implicit Differentiation: O(1) Memory

Rather than backpropagating through every iteration, gradients use implicit differentiation:

\frac{\partial L}{\partial \theta} = u^\top \frac{\partial T_{\theta_a}(\tilde{y}^\star, \tilde{y}_0)}{\partial \theta}, \quad u = (I - J_{\tilde{y}}^\top)^{-1} v

In practice, the authors use a one-step approximation $u \approx v$ , so backprop passes through the attractor only once (a single vector-Jacobian product). Training memory is ~4.18 GB regardless of whether the attractor iterates 2 or 64 times, while Parcae OOMs at 32–64 loops.

Anderson Acceleration

The fixed point is solved with Anderson acceleration (a quasi-Newton method), exiting when the residual norm falls below a tolerance $\varepsilon$ or a max iteration budget $T_{\max}$ is hit. Both are inference-time hyperparameters—tighten for accuracy, relax for speed, no retraining needed.

3. Experimental Results

Language Modeling (FineWeb-Edu): Pareto Improvement

| Scale | Model | Val. PPL | Lambada PPL | Core Accuracy | |:---|:---|:---|:---|:---| | 140M | Transformer | 21.48 | 127.39 | 13.00% | | | Parcae | 19.06 | 80.64 | 14.04% | | | Attractor | 18.30 | 68.02 | 14.59% | | 370M | Transformer | 15.79 | 40.77 | 17.46% | | | Parcae | 14.49 | 32.74 | 20.00% | | | Attractor | 14.03 | 27.14 | 20.24% | | 770M | Transformer | 13.08 | 22.37 | 22.42% | | | Parcae | 12.49 | 19.71 | 25.07% | | | Attractor | 12.09 | 15.21 | 26.83% | | 1.3B | Transformer | 11.95 | 17.26 | 25.45% |

Key numbers:

770M Attractor beats 1.3B Transformer—with the latter trained on 2x the data.
Lambada perplexity reduced 46.6% at 140M scale.
Core accuracy up 19.7% (relative) at 770M.
Training FLOPs 25–31% lower than Parcae.

Hard Reasoning Tasks

Sudoku-Extreme and Maze-Hard require single-pass direct answers with only ~1000 training examples:

| Method | Params | Sudoku-Extreme | Maze-Hard | |:---|:---|:---|:---| | Deepseek R1 | 671B | 0.0% | 0.0% | | Claude 3.7 | — | 0.0% | 0.0% | | O3-mini-high | — | 0.0% | 0.0% | | Transformer | 27M | 0.0% | 0.0% | | HRM | 27M | 55.0% | 74.5% | | TRM | 7M | 74.7% | 85.3% | | TRM | 27M | 0.0% (collapse) | 0.0% (collapse) | | Attractor | 7M | 54.3% | 46.7% | | Attractor | 27M | 91.4% | 93.1% |

Frontier LLMs and standard Transformers score 0% (the tasks forbid autoregressive decoding). TRM collapses when scaled to 27M; Attractor scales naturally. 27M parameters and 1000 examples beat the 671B DeepSeek R1—architecture fit beats parameter scale.

4. Equilibrium Internalization: The Most Striking Finding

After training, the backbone's initial output $\tilde{y}_0$ is already near the fixed point. At 770M scale, decoding the backbone output directly (T=0) is the *best* performance—the attractor barely needs to work at inference. By contrast, Parcae needs ~8 refinement loops to plateau.

The authors' interpretation: the attractor acts as a moving teacher during training. Early on, it does heavy refinement; later, the backbone learns to produce near-fixed-point guesses, internalizing the iterative process—like automatic knowledge distillation.

Practical implications:

1. The attractor can be removed at inference for speed, with minimal (or zero) performance loss. 2. Adaptive computation: simple inputs use T=0; hard inputs iterate more. 3. Model compression: the trained attractor can be distilled away, leaving a high-quality feedforward model.

5. Comparison with Related Work

| Method | Recurrence depth | Training memory | Inference adaptivity | Scalability | Stability | |:---|:---|:---|:---|:---|:---| | Standard Transformer | 1 | O(1) | ❌ | ✅ | ✅ | | Universal/Looped Transformer | fixed | O(T) | ❌ | ⚠️ | ⚠️ | | Parcae | fixed | O(T) | ❌ | ⚠️ | ✅ | | DEQ | adaptive | O(1) | ✅ | ⚠️ | ⚠️ | | Attractor Models | adaptive | O(1) | ✅ | ✅ | ✅ |

Ablations (60.3M params, 1B tokens) show why Attractor beats DEQ (34.05 vs 42.18 val PPL, 8.4 vs 14.6 avg iterations):

1. Fixed point in output-embedding space, not hidden state—closer to the prediction target. 2. Warm start from the backbone's guess instead of a zero vector. 3. Persistent injection anchors every iteration.

6. Author's Assessment

Why It Matters

1. O(1) memory training frees recurrent architectures from their memory bottleneck. 2. Equilibrium internalization shows recurrent refinement can be learned into a feedforward network—"train recurrent, infer feedforward." 3. Reasoning dominance: 27M beating 671B proves architecture-task fit matters more than parameter scale.

Limitations and Open Questions

1. The paper doesn't systematically study how large the attractor module should be. 2. Anderson acceleration's history windows and least-squares solves are unfriendly to hardware parallelism. 3. No inference-latency numbers: O(1) training memory ≠ fast inference. 4. Only validated on language modeling; multimodal fixed-point refinement is unknown. 5. Relationship to Chain-of-Thought (latent-space vs discrete-token iteration) is unexplored.

Impact Predictions

Short term (6–12 months): open-source replication; attractive for resource-constrained labs and startups.
Medium term (1–2 years): possible integration into mainstream frameworks (Hugging Face Transformers, vLLM); "train recurrent, infer feedforward" may become a standard paradigm; new hardware optimizations for fixed-point solving.
Long term (2–5 years): if equilibrium internalization proves general, model compression may shift from "distill big into small" to "train recurrent, naturally obtain high-quality feedforward models."

7. References

arXiv: 2605.12466 — Solve the Loop: Attractor Models for Language and Reasoning
Project page: https://attractor-models.github.io/
Code: https://github.com/jacobfa/Attractor
Related work: DEQ (Bai et al., 2019), Parcae (2026), Universal Transformers (Dehghani et al., 2019), Looped Transformers (Giannou et al., 2023a; Yang et al., 2024a), TRM/HRM (Jolicoeur-Martineau, 2025)
Datasets: FineWeb-Edu, Lambada, Core, Sudoku-Extreme / Maze-Hard (Wang et al., 2025)
Training framework: nanochat (Karpathy, 2025)

> Note: The 46.6% perplexity reduction, 770M-beats-1.3B, and 27M-beats-671B claims come from the paper's experiments. "Architecture fit beats parameter scale" and the teacher-student reading of equilibrium internalization are the original poster's analysis, not the paper's words.

*This article is based on arXiv 2605.12466 and public materials; interpretive views belong to the poster and do not represent the paper's authors or USC.*