Ouro: A 2.6B-Parameter Looped Language Model That Rivals 8B Transformers via Latent Recurrence
> Paper: Scaling Latent Reasoning via Looped Language Models > Authors: Zixuan Wang, Kai Hua, Tianyu Zhang, Ziniu Li, Haoran Que, et al. (ByteDance Seed, UC Santa Cruz, Princeton, Mila, CMU, UPenn, etc.) > arXiv: https://arxiv.org/abs/2510.25741 > Project page: https://ouro-llm.github.io/ > Code: https://github.com/rkstgr/LoopLM
Key points
- Core idea: Rather than scaling parameters or data, Ouro scales *iterative latent depth* — reapplying the same Transformer block multiple times to refine hidden representations. This targets the data-wall problem: high-quality human training data is finite, so decoupling compute from data is attractive.
- Architecture: Ouro-1.4B and Ouro-2.6B each use 24 layers looped 4 times (vs. 48 distinct layers in comparable 4B/8B standard Transformers), trained on 7.7T tokens. Reasoning happens in latent space instead of token space, avoiding the inefficiency of per-step vocabulary softmax.
- Adaptive dynamic exit: Each loop step has an Exit Gate computing an exit probability λₜ = sigmoid(Linear(hₜ)); a cumulative CDF threshold q controls the compute–accuracy trade-off at deployment (low q = faster, high q = more accurate).
- Reward hacking and entropy regularization: Naive training collapses all exit probability to the final step (a self-reinforcing loop since deeper loops yield lower loss). Adding an entropy term with a uniform prior over loop depths gives lower training loss and cleaner convergence than a geometric prior — the model learns which inputs need deeper reasoning on its own.
- Two-stage training: Stage I jointly pretrains the LM and exit gate with entropy regularization; Stage II freezes the LM and fine-tunes only the exit gate for the compute–accuracy trade-off.
- KV cache reuse: Full per-loop caching costs 4x memory. Reusing only the *first* loop's cache collapses performance (GSM8K 18.73 vs. 78.92 baseline), but reusing the *last* loop's cache is nearly lossless (78.62). Deployment can thus match standard-Transformer memory footprint.
- Upcycling pipeline: Ouro-1.4B is pretrained on 3T tokens; non-embedding layers are duplicated 2x to reach 2.6B parameters, then training continues on progressively higher-quality data up to 7.7T tokens total.
- Results: Ouro-1.4B (R4) matches a 4B standard Transformer and Ouro-2.6B (R4) rivals 8B models across math (GSM8K, MATH), science QA (MMLU), and code benchmarks — roughly 2–3x parameter efficiency. Excess looping can hurt performance in some settings, and more loops improve safety on HEx-PHI, including extrapolation beyond training-time loop counts.
- Physics-of-LLMs analysis: Looping does not increase knowledge storage (~2 bits/parameter either way), but substantially boosts knowledge manipulation — fact composition and multi-hop reasoning. In other words, looping makes the model smarter, not more knowledgeable. Latent iterative reasoning also produces more causally faithful reasoning traces than standard CoT, which can amount to post-hoc rationalization.
- Tested up to R4 loops; deeper extrapolation (R8, R16) remains to be validated.
- Entropy regularization requires careful β tuning; prefill cannot share KV caches, only decoding.
- A uniform exit prior may be suboptimal for tasks that never need deep loops.
- Future directions: conditional/hierarchical looping, combining loops with MoE or state-space models (e.g., Mamba), and edge/real-time inference via dynamic exit.
- Wang, Z., Hua, K., Zhang, T., Li, Z., Que, H., et al. (2025). *Scaling Latent Reasoning via Looped Language Models.* arXiv:2510.25741. https://arxiv.org/abs/2510.25741
- Project page: https://ouro-llm.github.io/
- Code: https://github.com/rkstgr/LoopLM
Limitations and future work
Significance
Ouro establishes iterative latent computation as a third scaling axis orthogonal to parameters and data, aligned with test-time compute scaling (OpenAI o1/o3-style reasoning) but realized at the architecture level. Intelligence may come not just from "more" (wider models) but from "deeper" (more iterations over the same weights).