A Six-Year-Old Idea That Never Took Off
In 2019, Google proposed the Universal Transformer — letting the same set of parameters process the input repeatedly, "thinking several rounds" before outputting, like a recurrent neural network. The theory was elegant: fewer parameters, iterative refinement, human-like repeated polishing of representations.
But six years on, the idea never really took off, for a simple and brutal reason: at equal pretraining compute, directly scaling model parameters by N usually beats looping the same layer N times.
For example, a 2.6B-parameter model looped 4 times costs the same pretraining compute as a one-pass 10.4B vanilla model — and the 10.4B vanilla model almost always wins.
This is the looped transformer's "compute accounting problem": the parameters you save are offset — and then some — by the extra compute you spend.
In July 2026, a paper from IQuest Research delivered a result that makes this architecture worth a second look: their Loopie-20B-A2B model, at matched pretraining compute, doesn't just avoid losing to the vanilla 30B-A3B — it beats it across the board. Even more striking, it won gold medals at the 2025 IMO (International Mathematical Olympiad) and IPhO (International Physics Olympiad) without any external tools.
Key Innovations: Layer-Loop + the Loopie Recipe
1. Layer-Loop: each layer loops itself, not the whole stack
Previous looped transformers mostly used stack-loop: run the entire transformer stack, then start over from the top. A three-layer model looped twice would run Layer1→Layer2→Layer3→Layer1→Layer2→Layer3.
Loopie uses layer-loop: each layer finishes its own loop before passing to the next. The same three-layer, two-pass schedule becomes Layer1→Layer1→Layer2→Layer2→Layer3→Layer3.
The difference seems subtle but is deep. Stack-loop requires information to be complete enough on the first pass through the whole stack, otherwise errors get amplified in the second loop. Layer-loop lets each layer polish its own hidden states before handing them on — more like local iterative optimization at every layer, rather than the whole network repeatedly trial-and-erroring.
2. The Loopie Recipe: a compute-matched scaling recipe
This is the paper's core contribution. Rather than dodging the "vanilla wins at equal compute" curse, the authors confront it directly: design a scaling recipe so the looped model can win even under compute-matched conditions.
The recipe balances three dimensions:
- Stored width: the model's hidden dimension
- Stored depth: the number of layers
- Recurrent depth: the number of loop iterations
Loopie-20B-A2B is the product of this recipe: 20B total parameters, 2B active parameters (MoE), with a two-pass loop. Its comparison target is the vanilla 30B-A3B (30B total, 3B active), with roughly equal pretraining compute.
Results: comprehensively beating the compute-matched vanilla baseline
Across multiple benchmarks, Loopie-20B-A2B outperforms the vanilla models with more parameters:
| Benchmark | Loopie-20B-A2B | Qwen3-30B-A3B-Thinking | gpt-oss-20B-High | Nemotron-3-Nano-30B-A3B | |------|---------------|----------------------|-----------------|----------------------| | AIME | 92.10% | 90.10% | 88.33% | 85.00% | | AMC | 94.21% | 93.57% | 91.80% | 91.05% | | OlympiadBench | 80.50% | 81.20% | 70.03% | 76.68% | | IFEval | 84.72% | 70.64% | 79.21% | 77.05% | | ARC-Challenge | 93.52% | 92.42% | 91.96% | 93.86% | | MMLU-Redux | 83.61% | 83.40% | 82.54% | 83.89% |
What does AIME 92.10% mean? That's the American Invitational Mathematics Examination — not multiple choice, competition problems requiring exact answers. A 20B-parameter (2B active) model solved 92% of them without a calculator or web search.
The IMO and IPhO gold medals push the claim "looping = smarter" from benchmarks onto the real competition stage.
Why only loop twice?
A natural question: if looping helps, why not loop 3, 5, or 32 times? Huginn previously looped 32 times.
Loopie's answer: two passes are enough.
The authors found that going from 1 to 2 iterations yields a significant performance gain, but from 2 to 3 the marginal benefit is tiny while compute cost doubles. This matches human intuition — thinking a problem through twice usually catches oversights from the first pass, but ten passes aren't much better than two.
This also echoes the layer-loop philosophy: depth isn't stacked via loop count; each loop must do substantive local optimization. Two passes — the first builds an initial representation, the second corrects and refines — already suffice.
The deeper implication: a spring for parameter sharing?
Loopie's success isn't just "another new model" — it reopens a question that had been closed: has parameter sharing been undervalued?
The mainstream narrative of recent years: more parameters are better, and MoE is just about cheaper inference. Loopie says: no — with the same compute, fewer but more cleverly reused parameters can do better.
There is an echo of how the brain works. The brain doesn't win by raw neuron count — the human brain has about 86 billion neurons, the elephant 257 billion, yet human cognition far exceeds the elephant's. The key lies in repeated use and iterative processing by neural circuits. Loopie's layer-loop, in a sense, simulates this mechanism of "the same group of neurons repeatedly processing the same problem".
That said, Loopie has limitations. Its training pipeline is complex, requiring careful initialization and learning-rate scheduling. Whether the two-pass sweet spot generalizes to larger scales (100B+) remains unknown. But at minimum, it proves the looped transformer is not a dead end — as long as the compute accounting is done right, looping can win.
---
Paper: Loop the Loopies!
Models: Loopie-20B-A2B, Loopie-6B-A0.6B (repos: megatron-loopie / vllm-loopie)