Solve the Loop: Attractor Models for Language and Reasoning

Forum topic · 小凯 · 2026-05-14

Summary

This paper introduces Attractor Models, a new architecture in which a backbone module proposes output embeddings and an attractor module refines them by solving for a fixed point, with gradients computed via implicit differentiation. This design keeps training memory constant regardless of effective depth and lets the number of iterations be chosen adaptively by convergence, addressing instability and fixed-depth limitations of looped Transformers. Empirically, Attractor Models improve both large-scale language-model pretraining and small-model reasoning: perplexity drops by up to 46.6% versus standard and stabilized recurrent Transformers, downstream accuracy improves by up to 19.7%, and a 770M model outperforms a 1.3B Transformer trained on twice the tokens. On hard reasoning tasks, a 27M-parameter model trained with about 1,000 examples reaches 91.4% on Sudoku-Extreme and 93.1% on Maze-Hard, where frontier models like Claude and GPT o3 fail entirely. The authors also report a phenomenon called equilibrium internalization, where fixed-point training pulls initial embeddings close to equilibrium so the solver can be removed at inference with little degradation.

Paper Overview

Field: NLP Authors: Jacob Fein-Ashley, Paria Rashidinejad arXiv: 2605.12466

Background

Looped Transformers iteratively refine latent representations, offering a promising alternative to purely feed-forward computation and improving language modeling and reasoning. However, recurrent architectures remain unstable to train, costly to optimize and deploy, and constrained to small, fixed recurrence depths.

Method: Attractor Models

A backbone module first proposes output embeddings.
An attractor module refines them by solving for a fixed point.
Gradients are obtained through implicit differentiation, so training memory remains constant in effective depth.
The number of iterations is chosen adaptively by convergence.

Results

Attractor Models outperform existing models across two regimes — large-scale language-model pretraining and reasoning with tiny models:

Language modeling: a Pareto improvement over standard Transformers and stabilized recurrent models at scale, with perplexity reduced by up to 46.6% and downstream accuracy improved by up to 19.7%, while lowering training cost. Notably, a 770M Attractor Model outperforms a 1.3B Transformer trained on twice as many tokens.
Reasoning with tiny models: a model with only 27M parameters and roughly 1,000 examples achieves 91.4% accuracy on Sudoku-Extreme and 93.1% on Maze-Hard — tasks on which frontier models such as Claude and GPT o3 fail completely, and specialized recursive reasoners collapse at larger scales.

Equilibrium Internalization

The paper also demonstrates a new phenomenon called equilibrium internalization: fixed-point training makes the model's initial output embeddings close to equilibrium, allowing the solver to be removed at inference time with minimal degradation.

---

*Auto-collected on 2026-05-14.*