Solve the Loop: Attractor Models for Language and Reasoning

Forum topic · 小凯 · 2026-05-14

Summary

Attractor Models, introduced by Jacob Fein-Ashley and Paria Rashidinejad (arXiv:2605.12466), address key limitations of looped Transformers for language modeling and reasoning. A backbone module first proposes output embeddings, and an attractor module refines them by solving for a fixed point, with gradients obtained via implicit differentiation. This design keeps training memory constant regardless of effective depth, while the number of iterations is selected adaptively by convergence. Empirically, Attractor Models deliver Pareto improvements over standard Transformers and stabilized recurrent models at scale: perplexity reductions up to 46.6%, downstream accuracy gains up to 19.7%, and lower training cost, with a 770M model outperforming a 1.3B Transformer trained on twice the tokens. On challenging reasoning tasks, a 27M-parameter model with roughly 1,000 examples achieves 91.4% on Sudoku-Extreme and 93.1% on Maze-Hard, tasks where frontier models like Claude and GPT o3 fail entirely. The paper also reports a new phenomenon, equilibrium internalization, where fixed-point training moves initial output embeddings near equilibrium so the solver can be removed at inference with minimal degradation.

Paper Overview

Field: NLP Authors: Jacob Fein-Ashley, Paria Rashidinejad arXiv: 2605.12466

Motivation

Looped Transformers offer a promising alternative to purely feed-forward computation by iteratively refining latent representations, improving language modeling and reasoning. However, recurrent architectures remain unstable to train, costly to optimize and deploy, and constrained to small, fixed recurrence depths.

Approach: Attractor Models

A backbone module first proposes output embeddings.
An attractor module then refines them by solving for the fixed point.
Gradients are obtained through implicit differentiation.
Training memory remains constant in effective depth, and iterations are chosen adaptively by convergence.

Results

Attractor Models outperform existing models across two regimes: large-scale language-model pretraining and reasoning with tiny models.

Language modeling: Pareto improvements over standard Transformers and stabilized recurrent models at scale, with perplexity reduced by up to 46.6% and downstream accuracy improved by up to 19.7%, while lowering training cost. A 770M Attractor Model outperforms a 1.3B Transformer trained on twice as many tokens.
Reasoning with tiny models: with only 27M parameters and about 1,000 examples, the model reaches 91.4% accuracy on Sudoku-Extreme and 93.1% on Maze-Hard — tasks on which frontier models such as Claude and GPT o3 completely fail, and where specialized recursive reasoners collapse at larger scales.

Equilibrium Internalization

The authors report a new phenomenon: fixed-point training pushes the model's initial output embeddings near equilibrium, allowing the solver to be removed at inference time with minimal degradation.

---

*Auto-collected on 2026-05-14*