[论文] Solve the Loop: Attractor Models for Language and Reasoning

## 论文概要 **研究领域**: NLP **作者**: Jacob Fein-Ashley, Paria Rashidinejad **发布时间**: 2026-05-12 **arXiv**: [2605.12466](https://arxiv.org/abs/2605.12466) ## 中文摘要 循环 Transformer 通过迭代细化潜在表示提供纯前馈计算的有前景替代，改善语言建模和推理。但循环架构训练不稳定、优化和部署成本高，且受限于小的固定递归深度。我们引入 Attractor Models，其中主干模块首先提议输出嵌入，然后吸引器模块通过求解不动点来细化它们，梯度通过隐式微分获得。因此，训练内存保持恒定，与有效深度无关，迭代由收敛自适应选择。实证上，Attractor Models 在两个体系中优于现有模型：大规模语言模型预训练和微型模型推理。在语言建模中，Attractor Models 在规模上为标准 Transformer 和稳定循环模型提供帕累托改进，困惑度降低高达 46.6%，下游准确率提高高达 19.7%，同时降低训练成本。值得注意的是，770M Attractor Model 优于在两倍 token 上训练的 1.3B Transformer。在挑战性推理任务上，仅 27M 参数和约 1000 示例的模型在 Sudoku-Extreme 上实现 91.4% 准确率，在 Maze-Hard 上实现 93.1%，Claude 和 GPT o3 等前沿模型完全失败，专业递归推理器在更大规模上崩溃。最后，Attractor Models 展示"平衡内化"新现象：不动点训练使模型的初始输出嵌入接近平衡，允许求解器在推理时移除而退化很小。 ## 原文摘要 Looped Transformers offer a promising alternative to purely feed-forward computation by iteratively refining latent representations, improving language建模 and reasoning. Yet recurrent architectures remain unstable to train, costly to optimize and deploy, and constrained to small, fixed recurrence depths. We introduce Attractor Models, in which a backbone module first proposes output embeddings, then an attractor module refines them by solving for the fixed point, with gradients obtained through implicit differentiation. Thus, training memory remains constant in effective depth, and iterations are chosen adaptively by convergence. Empirically, Attractor Models outperform existing models across two regimes, large-scale language-model pretraining and reasoning with tiny models. In language mod... --- *自动采集于 2026-05-14* #论文 #arXiv #NLP #小凯