LLMs Need More Loops, Not More Parameters: How Ouro's 2.6B Model Matches 8B Transformers
> Paper: Scaling Latent Reasoning via Looped Language Models > Authors: Zixuan Wang, Kai Hua, Tianyu Zhang, Ziniu Li, Haoran Que, et al. (ByteDance Seed, UC Santa Cruz, Princeton, Mila, CMU, UPenn, and others) > arXiv: https://arxiv.org/abs/2510.25741 > Project page: https://ouro-llm.github.io/ > GitHub: https://github.com/rkstgr/LoopLM
Key points
Why loop? The data wall problem
Scaling Laws (OpenAI, 2020) drove the race from GPT-3's 175B to trillion-parameter models, but high-quality human training data is finite. Ouro asks: can we decouple capability from data and parameters? Its answer: reuse computation instead of adding parameters — loop the same Transformer block to reason deeper in latent space rather than in token space.
Architecture: same block, applied repeatedly
| | Standard Transformer | Ouro (LoopLM) | |---|---|---| | Architecture | 48 distinct layers | 24 layers, looped 4x | | Parameters | 4B / 8B | 1.4B / 2.6B | | Reasoning space | Vocabulary (token-by-token) | Latent space | | Reasoning depth | Fixed by layer count | Adaptive via exit gate | | Training data | Trillions of tokens | 7.7T tokens |
Each recurrent pass refines the hidden state, analogous to re-thinking a problem more deeply. An exit gate computes a per-step exit probability (CDF over loops); a threshold hyperparameter q trades speed vs. accuracy at deployment.
Fixing reward hacking with entropy regularization
Naively trained exits collapse to the maximum loop depth (deeper loops lower loss, creating a self-reinforcing bias). Adding an entropy regularizer with a uniform prior over exit depths prevents this: all depths get training signal, and the model learns which inputs need deeper reasoning. Training happens in two stages — joint pretraining (LM + exit gate), then exit-gate-only fine-tuning with the LM frozen.
KV cache sharing: memory-efficient deployment
Looping naively costs 4x KV cache memory. Tested strategies on GSM8K:
- Full per-step cache (baseline): 78.92
- First-step reuse: 18.73 (collapse)
- Final-step reuse: 78.62 (near-lossless)
- Averaged cache: 78.33
- Ouro-1.4B (R4) matches standard 4B Transformers
- Ouro-2.6B (R4) matches standard 8B Transformers
- Consistent gains on GSM8K, MATH, MMLU, and code benchmarks vs. same-size models
- Over-looping can hurt some tasks (analogous to analysis paralysis)
- Safety improves with more loops on the HEx-PHI harmfulness benchmark, including extrapolation beyond training loop counts
- Knowledge storage: unchanged (~2 bits/parameter, same as non-looped models)
- Knowledge manipulation: significantly improved — fact composition and multi-hop reasoning
- Faithfulness: iterative latent updates produce reasoning traces more causally aligned with outputs than standard CoT, which often acts as post-hoc rationalization
- Wang, Z., et al. (2025). Scaling Latent Reasoning via Looped Language Models. arXiv:2510.25741
- Project page: https://ouro-llm.github.io/
- Code: https://github.com/rkstgr/LoopLM
Final-step representations are rich enough to support decoding, so Ouro runs with roughly standard Transformer memory while providing 4x latent depth.
Training pipeline: upcycling 1.4B → 2.6B
Ouro-1.4B (24 layers) is pretrained on 3T tokens; non-embedding layers are duplicated 2x to 48 layers (2.6B params), then trained further on higher-quality data (7.7T total), followed by exit-gate fine-tuning.
Results: 2–3x parameter efficiency
Physics of LLMs: what loops actually do
In short: loops don't make the model more *knowledgeable*, they make it *smarter*.
Significance and limitations
The paper positions iterative latent depth as a third scaling axis beyond parameters and data, complementing test-time compute scaling (e.g., OpenAI o1/o3).
Limitations: only R4 tested in depth; entropy-regularization β requires careful tuning; KV sharing works only in the decode phase, not prefill. Future directions include deeper loops (R8, R16), conditional/hierarchical looping, combining loops with MoE or state-space models (e.g., Mamba), and edge/real-time inference via adaptive exits.
> "This work establishes iterative latent computation as a critical third scaling axis beyond parameters and data."