Attractor Models From Zero to Mastery: Teaching AI to Think Iteratively

Forum topic · 小凯 · 2026-05-14

Summary

This Feynman-style tutorial from zhichai.net explains Attractor Models, based on the paper "Solve the Loop: Attractor Models for Language and Reasoning" (arXiv:2605.12466) with code at https://github.com/jacobfa/Attractor. It explains why standard Transformers' fixed-depth, single-pass processing fails at complex reasoning, and how recurrent Transformers refine representations iteratively. It covers the three classic pitfalls—exploding training memory, vanishing/exploding gradients, and fixed-loop depth—and how fixed-point solving addresses them. Key concepts include fixed points (states where iteration stabilizes), implicit differentiation that gives O(1) memory via a one-step vector-Jacobian approximation, and Anderson acceleration for faster convergence (average iterations reduced from 14.6 to 8.4). The architecture pairs a large backbone Transformer that produces an initial guess with a small attractor module that iterates to a fixed point, with persistent injection of the initial guess at every step. The most striking finding is "equilibrium internalization": after training, the backbone alone matches full attractor-augmented performance (at 770M scale, no attractor is needed at inference), enabling training with loops but feedforward inference. The article includes runnable PyTorch implementations, when-to-use guidance, and common misconceptions, noting reported results such as 770M beating 1.3B and 27M beating 671B.

This is a Feynman-style tutorial (from zero, with runnable code) explaining Attractor Models, based on the paper *Solve the Loop: Attractor Models for Language and Reasoning* (arXiv:2605.12466). Code: https://github.com/jacobfa/Attractor · Project page: https://attractor-models.github.io/

Key points

Why recurrence matters: Humans understand iteratively (first pass: what; second: how; third: why). Standard Transformers are like a row of snapshot cameras—each layer fires once, with no chance to "rethink," which limits complex reasoning.
Recurrent Transformers reapply the same layer T times, but suffer three pitfalls: (1) training memory explodes linearly with loop count, (2) gradients explode/vanish over long loops, (3) inference depth is bound to the trained loop count.
Fixed points solve adaptivity: iterate until y* = f(y*), i.e., the state stops changing—the model itself decides when it has "thought enough."
O(1) memory via implicit differentiation: instead of storing all intermediate states, use the implicit function theorem: ∂y*/∂θ = (I − ∂f/∂y*)^(−1) · ∂f/∂θ. With a one-step approximation (u ≈ v, one vector-Jacobian product through the attractor), backprop costs the same whether the forward pass iterates 2 or 20 steps.
Anderson acceleration uses a history of past residuals to extrapolate toward the fixed point, converging ~2–3× faster; the paper's average iterations drop from 14.6 (DEQ) to 8.4.

Architecture: backbone + attractor

A large backbone Transformer generates a good-but-rough initial guess y₀ (the "first-pass understanding").
A small attractor module iterates that guess to the fixed point y* (the "careful re-thinking").
Persistent injection: every attractor step re-injects y₀ (e.g., y = y_t + 0.1 * y_0), preventing drift toward irrelevant attractors.

Equilibrium internalization (the most striking finding)

During training, the backbone gradually learns to output nearly fixed-point-quality guesses so the attractor has little work left. Result: at 770M scale, the backbone alone (no attractor, T=0) already achieves best performance. This enables train with loops, infer feedforward—with the attractor available on demand for extra precision. Reported headline results from the paper: 770M beating 1.3B, and 27M beating 671B on reasoning tasks.

Minimal implementation sketch

The tutorial provides a runnable PyTorch AttractorModel: shared embeddings, a backbone TransformerEncoder producing y₀, a smaller attractor encoder iterated until max(abs(y_new − y)) < tol (with persistent injection), and an unembedding head. Fixed-point solving can be skipped at inference (use_attractor=False). True training requires implicit differentiation (the tutorial shows a phantom-gradient sketch and recommends the torchdeq library).

When to use Attractor Models

⭐⭐⭐⭐⭐ Iterative-reasoning tasks (Sudoku, mazes, math proofs) and memory-constrained pretraining (O(1) memory allows deep effective loops)
⭐⭐⭐⭐ Long-context understanding (e.g., Lambada-style long-range dependencies)
⭐⭐⭐ Low-latency inference (post-training backbone-only, after validating latency)
⭐⭐ Standard NLP tasks (likely overkill) and multimodal (untested in the paper)

Common misconceptions

1. "Recurrence = slow" — inference can be fast via internalization; iteration count is adaptive (simple inputs converge in ~2 steps). 2. "Bigger attractor is better" — it should be small and focused on refinement, not global understanding. 3. "A fixed point means a correct answer" — it only means iteration stabilized; quality still depends on training. 4. "Implicit differentiation is too hard" — mature libraries like torchdeq exist.

Caveats

All analogies (roly-poly toy, snapshots, teacher–student) are pedagogical and break at mathematical edges (e.g., attractors may have multiple fixed points). Experimental figures come from arXiv:2605.12466; code is simplified for teaching and is not the official implementation. Prerequisites: standard Transformers, RNN basics, and Deep Equilibrium Models (Bai et al., 2019). Further reading: Parcae (2026), cited as the current state-of-the-art recurrent language model.