Dual-Path Architecture Lets LLMs Choose Between Depth and Width Per Token

Forum topic · ✨步子哥 · 2026-05-31

Summary

A Dual-Path Block architecture proposed by researchers at the Lamarr Institute, Fraunhofer IAIS, and University of Bonn resolves the trade-off between looped (deep) and standard (wide) Transformers in LLMs. Each layer contains two parallel paths: a deep path with a parameter-shared block iterated N times, suited to multi-step reasoning, and a wide path with an enlarged feed-forward network, suited to knowledge retrieval. A per-token sigmoid gate computes weights to mix the two outputs, letting every token independently choose its compute allocation. Gating weights are directly interpretable: function words and lexical content route toward the wide path, while punctuation and mathematical/symbolic tokens route toward the deep path—emerging naturally from training rather than hand-crafted rules. At matched FLOP budgets of roughly 1.5B and 3B parameters, Dual-Path models outperform equal-compute baselines on language modeling and downstream tasks while using fewer parameters. Ablations confirm both paths are necessary, that per-token gating matters (random permutation degrades performance), and that loop counts do not extrapolate beyond the training regime. The work suggests per-token compute allocation as a promising direction for next-generation LLM architectures, though validation remains at smaller scale.

Source

A Dual-Path Architecture for Scaling Compute and Capacity in LLMs — Markus Frey, Behzad Shomali, Joachim Koehler, Mehdi Ali (Lamarr Institute / Fraunhofer IAIS / University of Bonn): https://arxiv.org/abs/2605.30202

The problem: deep vs. wide

Standard Transformers treat every token identically, applying the same depth and parameters whether processing a function word or a math expression. Looped Transformers compensate with depth—repeating a shared block to save parameters at the cost of compute—while standard Transformers go wide, stacking parameters for one-shot capacity at the cost of memory. The paper asks: why choose one over the other?

Dual-Path Block

Each layer runs two parallel paths on the same input:

Deep Path: a parameter-shared Transformer block looped N times. Few parameters, repeated "thinking"—suited to tokens needing multi-step reasoning.
Wide Path: a single pass through a block with an enlarged FFN. More parameters, one pass—suited to tokens needing broad knowledge retrieval.

A per-token gate computes two sigmoid weights $$g_d$$ and $$g_w$$ independently for each token, producing the final output $g_w \cdot h_{\text{wide}} + g_d \cdot h_{\text{deep}}$ . In effect, each token decides for itself whether it needs deeper iteration or wider lookup.

Interpretable gating

The learned gate distributions show a clear, directly interpretable pattern:

| Token type | Gate preference | Intuition | |-----------|-----------------|-----------| | Function words | → Wide path | Language knowledge, no reasoning needed | | Lexical content (verbs, adjectives) | → Wide path | Semantic knowledge retrieval | | Punctuation | → Deep path | Structural markers, multi-step handling | | Math/symbols | → Deep path | Multi-step reasoning |

This division of labor emerged naturally during training, mirroring the human reading pattern where function words are skimmed and formulas are pondered.

Results

At two FLOP budgets (~1.5B and 3B parameter scale), Dual-Path models outperform equal-compute baselines on language modeling and downstream evaluations while using fewer parameters. Ablations show:

Both paths are indispensable: disabling either ( $$g_d=0$$ or $$g_w=0$$ ) significantly hurts performance.
Per-token decisions matter: randomly permuting gate weights (preserving marginal distributions) degrades performance.
Loop counts do not extrapolate: training with 3 loops does not benefit from 5 loops at inference—the learned loop dynamics have a comfort zone.

Why it matters

Dual-Path dissolves a long-standing architectural dilemma: looped Transformers save parameters but lack capacity; standard Transformers have capacity but are parameter-hungry. The answer is to provide both and let the model allocate per token. The broader implication is that per-token compute allocation may be key to next-generation LLM architectures—with a simple interpretable gate sufficing to learn when to think hard and when to skim.

Limitations: validation is limited to smaller scales (1.5B–3B), and whether a deep/wide binary is the optimal allocation granularity remains open.

Paper: https://arxiv.org/abs/2605.30202