Loopie-20B-A2B: Looped Transformer Finally Beats Compute-Matched Vanilla Models

Forum topic · ✨步子哥 · 2026-07-20

Summary

Looped Transformers have long lost to vanilla models under equal pre-training compute: a 2.6B model looped 4 times matches the compute of a 10.4B vanilla model, which usually wins. A July 2026 paper from IQuest Research presents Loopie-20B-A2B, a 20B-total-parameter MoE model (2B active) that loops each layer twice, reversing this trend. Key innovations are layer-loop—each layer iterates before passing activations onward, instead of looping the whole stack—and a compute-matched scaling recipe aligning effective depth (stored depth × recurrent depth) rather than raw parameter count. Loopie-20B-A2B outperforms Qwen3-30B-A3B-Thinking and gpt-oss-20B-High on AIME (92.10%), AMC, OlympiadBench, and IFEval, and won gold at IMO 2025 and IPhO 2025 without external tools. The authors find two loops is the sweet spot: going from one to two loops yields large gains, while more loops bring diminishing returns at doubled cost. The result reopens the question of whether parameter sharing has been undervalued.

A Six-Year-Old Idea That Never Took Off

In 2019, Google proposed the Universal Transformer—letting the same set of parameters process the input repeatedly, "thinking" over multiple passes like an RNN. In theory it was elegant: fewer parameters, iterative refinement.

But the idea never gained traction for a simple, brutal reason: under equal pre-training compute, scaling parameters N× almost always beats looping the same layer N times. A 2.6B model looped 4 times costs the same pre-training compute as a vanilla 10.4B model—and the vanilla model nearly always won. This is the "compute accounting problem" of looped Transformers: parameter savings get cancelled out (and then some) by extra compute.

In July 2026, a paper from IQuest Research delivered a result worth re-examining: their Loopie-20B-A2B model, at matched pre-training compute, not only matched but comprehensively beat vanilla 30B-A3B. More strikingly, it won gold at IMO and IPhO 2025 without any external tools.

Key Innovations: Layer-Loop + Loopie Recipe

1. Layer-Loop instead of Stack-Loop

Previous looped Transformers mostly used stack-loop: run the entire stack, then repeat (Layer1→Layer2→Layer3→Layer1→Layer2→Layer3).

Loopie uses layer-loop: each layer iterates before passing on (Layer1→Layer1→Layer2→Layer2→Layer3→Layer3).

The difference is subtle but deep. Stack-loop requires information to be nearly complete on the first pass, or errors get amplified on the second. Layer-loop lets each layer polish its hidden state before handing it on—like local iterative optimization per layer, rather than the whole network repeatedly retrying.

2. The Loopie Recipe: Compute-Matched Scaling

The core contribution is a scaling recipe designed to let looped models win under compute-matched conditions. It considers three dimensions:

Stored width: hidden dimension
Stored depth: number of layers
Recurrent depth: loop count

The key insight: a looped model should not align parameter count with the vanilla model, but should align effective depth (stored depth × recurrent depth), using MoE to maximize parameter efficiency.

Loopie-20B-A2B is the product of this recipe: 20B total parameters, 2B active (MoE), two loop iterations—compared against vanilla 30B-A3B (30B total, 3B active) at roughly equal pre-training compute.

Results: Beating the Compute-Matched Vanilla Baseline

| Benchmark | Loopie-20B-A2B | Qwen3-30B-A3B-Thinking | gpt-oss-20B-High | Nemotron-3-Nano-30B-A3B | |------|---------------|----------------------|-----------------|----------------------| | AIME | 92.10% | 90.10% | 88.33% | 85.00% | | AMC | 94.21% | 93.57% | 91.80% | 91.05% | | OlympiadBench | 80.50% | 81.20% | 70.03% | 76.68% | | IFEval | 84.72% | 70.64% | 79.21% | 77.05% | | ARC-Challenge | 93.52% | 92.42% | 91.96% | 93.86% | | MMLU-Redux | 83.61% | 83.40% | 82.54% | 83.89% |

AIME 92.10% means solving 92% of AIME problems—exact-answer competition math, no calculator, no web search—for a 20B-total (2B-active) model. The IMO and IPhO golds push "looping = smarter" from benchmarks onto the real competition stage.

Why Only Two Loops?

Why not loop 3, 5, or 32 times (as Huginn did)? Loopie's answer: two is enough. Going from 1 to 2 loops yields significant gains; from 2 to 3, marginal returns shrink while compute doubles. This matches the layer-loop philosophy: depth comes not from stacking loops, but from each iteration doing substantive local refinement—the first pass builds an initial representation, the second corrects and refines it.

Broader Implications: A Spring for Parameter Sharing?

Loopie reopens a closed question: has parameter sharing been undervalued? The mainstream narrative has been "more parameters is better; MoE just makes inference cheaper." Loopie says no—at the same compute, fewer but smarter-reused parameters can do better.

There's an analogy to the brain: humans have ~86 billion neurons versus elephants' 257 billion, yet far surpass elephant cognition—the key is repeated use and iterative processing of neural circuits. Layer-loop, in a sense, simulates "the same neurons processing the same problem repeatedly."

Limitations remain: the training pipeline is complex (careful initialization and LR scheduling), and whether the two-loop sweet spot extends to 100B+ scale is unknown. But Loopie proves looped Transformers are not a dead end—with correct compute accounting, looping can win.

---

Paper: Loop the Loopies!

Models: Loopie-20B-A2B, Loopie-6B-A0.6B (repos: megatron-loopie / vllm-loopie)