Loop Scaling Laws: Halving Parameters Can Still Beat Bigger MoE Models — But the Math Is Nuanced

In late September, three Meta researchers posted a paper (arXiv 2609.40316) asking a deceptively engineering-flavored question: how large should a model be (total parameters), how much should it activate per token (sparsity), and how many times should the same layers be rerun (loop count)? Previously these were accounted separately — MoE had its scaling laws, recurrence had its own — and nobody had combined them. This work, *Loop Scaling Laws*, delivers the first joint scaling law.

The headline that spread: "parameters cut in half, reasoning performance actually improves." Worth unpacking, because the unpacked version is more interesting.

What the paper actually measured

Trained on one trillion tokens at equal training compute:

• LoopMoE: 0.3B active, 1.3B total parameters
• Baseline MoE: 0.6B active, 2.9B total (2.2x the total params, 2x the active params)

Results:

• At R=1 (no looping), the smaller model used only 0.4x the inference compute and matched the baseline.
• The win came at R=5: 10.3 points higher, at the cost of 1.8x the inference compute.

So "not dropping, but rising" is, strictly speaking, not a parameter-efficiency achievement — it's a test-time-compute achievement. Halving the parameters bought parity; the extra 10.3 points were bought with four extra loops. But the unpacked accounting is more valuable: it proves looping is a real knob, not decoration. The same model matches a larger model on the cheap setting and beats it on the deep-thinking setting.

The ceiling problem

Looping isn't new — Universal Transformers (2018) tried applying the same transformer layers repeatedly, with follow-ups since. It never went mainstream because loop gains saturate: two loops help, ten loops yield near zero. This paper formalizes that ceiling into the scaling law (a bounded loop-gain mapping), and finds that sparsity raises the ceiling: MoE's dynamic routing sends tokens to different experts each loop, so repeated loops perform genuinely different computation. The precise version of "looping hits a wall, MoE pushes it open" is: the wall is still there, but sparsity moves it further away.

The marginal-returns table

The paper's table shows what the raised ceiling looks like. For the same model, going from R=1 to R=5 yields gains of +6.4, +3.0, +0.7, +0.2 per additional loop — each loop's marginal return shrinks. This curve deserves a look from anyone talking about "deep thinking": thinking harder isn't free, and the fourth loop is almost worthless.

The efficiency ledger

Two headline numbers: sparsity provides roughly 3x active-parameter efficiency, looping roughly 2x total-parameter efficiency (on reasoning tasks). The two axes are complementary and stack: a 0.3B-active model cranked to R=5 beats a 0.6B-active opponent.

The bigger picture: a third knob

In the broader landscape, the paper adds a third knob to the MoE era. Recent flagship models have all shrunk active parameters — GLM at 320B-A18B, DeepSeek at 552B-A8B: the activation ratio is the interface, total parameters are a capacity commitment, and what you actually use per call is the bill. LoopMoE adds a third knob: total params govern capacity, active params govern per-use cost, loops govern how many times you rerun what you use. "Deep thinking mode" was previously rhetoric — longer chains of thought, more sampling; now it has a parameter-level form: rerun the same layers more times, adjustable at inference, with the same model offering both a cheap mode and a deep-thinking mode.

The real contribution

One trillion training tokens, matching a 2.2x-larger model at 0.4x compute, beating it by 10.3 points at 1.8x compute — behind all these numbers, the genuinely new thing is that marginal-returns table: +6.4, +3.0, +0.7, +0.2. Each loop of "thinking more" is cheaper than the last, and also less useful than the last. The marginal-cost curve of deep thinking has, for the first time, been drawn by a scaling law.

---

Source notes: Paper arXiv 2609.40316, *Scaling Laws for Looped Mixture of Experts* (2026-09-30, Yanbei Chen, Anirudh Goyal, Raghuraman Krishnamoorthi, Meta AI); abstract and full text verified. Figures (3x active-parameter efficiency, 2x total-parameter efficiency, 0.3B/1.3B vs 0.6B/2.9B, R=1 to 5, +10.3 points, 0.4x inference compute at R=1, 1.8x at R=5, 2.2x total params, one trillion training tokens) all from the paper body and Table 2. Attribution ruling: parity belongs to parameter efficiency; the win belongs to test-time compute (R=1→5) at 1.8x inference compute. Universal Transformers (Dehghani et al., 2018) provides the looped-architecture lineage.