Dual-Path Architecture Lets LLMs Choose Between Depth and Width Per Token

Forum topic · ✨步子哥 · 2026-05-31

Summary

Researchers from the Lamarr Institute, Fraunhofer IAIS, and the University of Bonn propose a Dual-Path Block that resolves the trade-off between looped (deep) and standard (wide) Transformers. Each layer contains two parallel paths: a parameter-shared Transformer block looped N times for multi-step reasoning, and a single-pass block with an enlarged feed-forward network for knowledge retrieval. A per-token sigmoid gate computes weights g_d and g_w to blend both outputs, letting each token decide its own compute allocation. Gating weights are directly interpretable: function words and lexical content prefer the wide path, while punctuation and mathematical symbols prefer the deep path—emerging naturally during training without manual rules. At matched FLOP budgets (~1.5B and 3B parameters), Dual-Path outperforms compute-matched baselines with fewer parameters. Ablations confirm both paths are necessary, per-token gating matters, and loop counts do not extrapolate beyond training. The work suggests per-token compute allocation as a key direction for next-generation LLM architectures.

> Source: A Dual-Path Architecture for Scaling Compute and Capacity in LLMs, Markus Frey, Behzad Shomali, Joachim Koehler, Mehdi Ali, Lamarr Institute / Fraunhofer IAIS / University of Bonn, https://arxiv.org/abs/2605.30202

---

When you read an article, function words like "the" or "is" pass through your brain almost effortlessly, while a mathematical formula or chain of logical reasoning demands repeated, careful processing—the same reading task, with radically different "compute" allocated to different positions.

Large language models can't do this. Traditional Transformers treat every token identically, using the same network depth and the same parameters to process "the" and "∫". Looped Transformers try to compensate with depth—executing the same module repeatedly, saving parameters but spending compute; standard Transformers take the "wide" route—stacking more parameters for a one-shot pass, but costing memory.

The question is: why choose between deep and wide at all?

Dual-path: depth and width coexist

This paper proposes the Dual-Path Block, with a remarkably simple core idea: place two paths in parallel within the same layer—

Deep Path: a parameter-shared Transformer block executed N times in a loop. Few parameters, but it can "think" repeatedly—suited to tokens requiring multi-step reasoning.
Wide Path: a single-pass block with an enlarged feed-forward network (FFN). More parameters, but only traversed once—suited to tokens requiring extensive knowledge retrieval.

Both paths process the same input, and a Per-Token Gate decides how to blend them: each token independently computes two sigmoid weights $$g_d$$ and $$g_w$$ , and the final output is $g_w \cdot h_{\text{wide}} + g_d \cdot h_{\text{deep}}$ .

It's like giving the model a "compute dispatcher"—instead of forcing all tokens down the same path, each token chooses for itself: do you need to think several times, or consult more knowledge?

What did the gates learn? Directly interpretable

The most exciting finding: the gate weights are directly interpretable, requiring no post-hoc analysis.

The team examined the trained gating distributions and found a clear pattern:

| Token type | Gate preference | Intuition | |-----------|---------|---------| | Function words (the, is, of) | → Wide path | No reasoning needed, language knowledge needed | | Lexical content (verbs, adjectives) | → Wide path | Needs a semantic knowledge base | | Punctuation | → Deep path | Structural markers, multi-step processing | | Math/symbols | → Deep path | Needs multi-step reasoning |

This echoes human reading cognition: function words rely on "intuition" (wide path = large knowledge base in one step), while math symbols rely on "deliberation" (deep path = repeated derivation). The model wasn't forced into this division by hand-coded rules—the specialization emerged naturally during training.

Results: winning across the board at matched compute

At two FLOP budgets (~1.5B and 3B parameter scale), Dual-Path models outperformed compute-matched baselines on both language modeling and downstream task evaluations, while using fewer parameters.

The ablation studies are equally convincing:

Both paths are indispensable: disabling either one ( $$g_d=0$$ or $$g_w=0$$ ) significantly degrades performance.
Per-token decisions matter: randomly shuffling the gate weights (keeping marginal distributions fixed) also hurts performance, showing the model genuinely allocates compute on a token-by-token basis.
Loop counts don't extrapolate: training with 3 loops and increasing to 5 at inference doesn't help—the learned loop dynamics have a "comfort zone."

Why this matters

This paper resolves a long-standing architectural dilemma: looped Transformers save parameters but lack capacity, while standard Transformers have capacity but spend parameters. Dual-Path's answer: don't choose—have both, and let the model decide which path each token takes.

The deeper implication: per-token compute allocation may be a key direction for next-generation LLM architectures. The human brain doesn't spend equal effort on every word, and models shouldn't either. Dual-Path demonstrates this with the simplest possible mechanism—an interpretable gate is enough for the model to learn "when to deliberate, and when to skim."

That said, the work has limitations: it's validated only at smaller scale (1.5B–3B), and whether the deep/wide dichotomy is the optimal allocation granularity remains open. But the direction is right—teaching a model to tailor its effort per token is far more efficient than stewing everything in one pot.

---

Paper link: https://arxiv.org/abs/2605.30202