ARLoop: 1.2B Looped Transformer Beats 3.5B Models, First Real-Scale Success for Recurrent LLM Pretraining

ARLoop, developed by teams from Ohio State and Saarland University, marks the first time a looped Transformer has outperformed non-recurrent models at real pretraining scale: a 1.2B-parameter ARLoop trained on 0.5T tokens beats the 3.5B-parameter Huginn (the only previously fully open-source looped LLM) on all 8 benchmarks, and is competitive with Qwen3.5-2B, which uses more parameters and more training data.

Background: the problem with looped Transformers

A looped Transformer applies the same set of layers R times instead of adding parameters. The standard approach accumulates all intermediate representations from every loop into the residual stream ("full-history attention"), but not all historical representations are useful—some are noise, and the noise compounds with deeper loops.

ARLoop's two core innovations

1. Attention residuals instead of additive residuals. Standard residuals compute output = layer(input) + input. ARLoop computes output = Attention(layer(input), input), letting the model learn how much of each prior layer's output to keep via attention-weighted aggregation.

2. Sliding-window depth attention. Counterintuitively, at deeper loop counts, attending only to the most recent 2-3 loops outperforms full-history attention. Early-loop representations are too "coarse"—like having a college student reread elementary-school textbooks. The window keeps the model focused on its most recent, more refined thinking.

Controlled experiments before scaling

At 140M and 370M parameters with loop depth R=1 to 5, ARLoop consistently beats prior looped residual designs (PreNormLoop, Sandwich Norm, DeepLoop, Parcae) on language-modeling perplexity. Sliding-window attention matches full history at shallow loops and beats it at deep loops; attention weights confirm the model focuses on recent iterations.

Real-scale pretraining results

• ARLoop-1.2B beats Huginn-3.5B on all 8 benchmarks, with ~3x fewer parameters and less data.
• ARLoop-1.2B matches or beats Qwen3.5-2B on most of the 8 benchmarks.
• Largest gains on math and coding, where deeper loops contribute most—consistent with these tasks needing longer reasoning chains.
• ARLoop-0.6B beats the parameter-matched PreNormLoop baseline on 7 of 8 benchmarks, showing the gains are architectural, not scale-driven.

A "layer effect" analysis shows that on general tasks, the first loop contributes most correctness with diminishing returns afterward, while on math and coding, later loops build meaningfully on earlier ones—like listing equations, then solving, then verifying.

Implications

• Parameter efficiency: the looped Transformer's promise—equal performance with fewer parameters—is validated at real scale for the first time.
• Inference-time scaling: loop count is a controllable compute knob (1 loop for easy queries, 3-5 for hard ones). Sliding-window attention prevents degradation at deep loops.
• Architecture design: attention residuals and sliding windows are independent choices; the "deep loops don't need full history" finding may generalize to other architectures.

Limitations

• Only 0.5T training tokens; whether the advantage holds at larger scale is unknown.
• Loop gains diminish on general tasks; value is concentrated in math and coding.
• Inference latency grows linearly with loop count.

Fully open source

Training code, data processing code, intermediate checkpoints, and final model weights are all released.

Paper: https://arxiv.org/abs/2611.05234 Code and models: https://github.com/OSU-NLP-Group/ARLoop