Solve the Loop: Attractor Models for Language and Reasoning

Forum topic · 小凯 · 2026-05-14

Summary

Attractor Models, introduced by Jacob Fein-Ashley and Paria Rashidinejad (arXiv:2605.12466), combine a backbone module that proposes output embeddings with an attractor module that refines them by solving for a fixed point, with gradients obtained via implicit differentiation. This design keeps training memory constant regardless of effective depth and lets the number of iterations be chosen adaptively by convergence, addressing instability and cost issues of looped Transformers. In large-scale language-model pretraining, Attractor Models provide a Pareto improvement over standard Transformers and stabilized recurrent models, reducing perplexity by up to 46.6% and improving downstream accuracy by up to 19.7% at lower training cost; a 770M model outperforms a 1.3B Transformer trained on twice the tokens. On hard reasoning tasks, a 27M-parameter model with about 1,000 examples reaches 91.4% on Sudoku-Extreme and 93.1% on Maze-Hard, where frontier models like Claude and GPT o3 fail. The paper also reports "equilibrium internalization": fixed-point training moves initial embeddings near equilibrium, allowing the solver to be removed at inference with little degradation.

Paper Overview

Field: NLP
Authors: Jacob Fein-Ashley, Paria Rashidinejad
Published: 2026-05-12
arXiv: 2605.12466

Abstract

Looped Transformers offer a promising alternative to purely feed-forward computation by iteratively refining latent representations, improving language modeling and reasoning. Yet recurrent architectures remain unstable to train, costly to optimize and deploy, and constrained to small, fixed recurrence depths.

The authors introduce Attractor Models, in which a backbone module first proposes output embeddings, then an attractor module refines them by solving for the fixed point, with gradients obtained through implicit differentiation. As a result:

Training memory remains constant in effective depth.
The number of iterations is chosen adaptively by convergence.

Empirical Results

Attractor Models outperform existing models across two regimes: large-scale language-model pretraining and reasoning with tiny models.

Language modeling:

Pareto improvement over standard Transformers and stabilized recurrent models at scale
Perplexity reduced by up to 46.6%
Downstream accuracy improved by up to 19.7%
Lower training cost; a 770M Attractor Model outperforms a 1.3B Transformer trained on twice as many tokens

Reasoning:

With only 27M parameters and about 1,000 examples: 91.4% accuracy on Sudoku-Extreme and 93.1% on Maze-Hard
Frontier models (Claude, GPT o3) fail outright on these tasks, and specialized recursive reasoners collapse at larger scale

Equilibrium Internalization

The paper reports a new phenomenon, *equilibrium internalization*: fixed-point training moves the model's initial output embeddings close to equilibrium, allowing the solver to be removed at inference time with little degradation.

---

*Auto-collected on 2026-05-14*