mHC: Manifold-Constrained Hyper-Connections (Xie et al., 2025, DeepSeek)

Forum topic · 小凯 · 2026-05-10

Summary

mHC (Manifold-Constrained Hyper-Connections), a 2025 paper by Xie et al. from DeepSeek (arXiv: 2512.24880), addresses a key weakness of Hyper-Connections (HC): the loss of ResNet's identity mapping property, which guarantees network output equals input when all sublayer outputs are zero. This property matters for gradient propagation and initialization stability, and HC breaks it whenever the dynamic residual weights alpha_l are not equal to 1. mHC projects the HC residual space onto a manifold of parameter configurations that satisfy the identity mapping property, constraining parameter updates to this manifold. This preserves dynamic residual reweighting while restoring stability for large-scale training of deep networks (100+ layers). The authors also perform infrastructure optimizations (memory access and communication) to keep the overhead of dynamic connections manageable. Reported results include tangible performance improvements, superior scalability, and resolution of HC's severe training instability. The forum post frames mHC as the theoretical completion of HC, illustrating a break-and-repair pattern in deep architecture evolution.

mHC: Manifold-Constrained Hyper-Connections (2025, Xie et al., DeepSeek)

arXiv: 2512.24880

Core problem

HC's dynamic residual weighting works well, but it destroys an important property of ResNet: the identity mapping. In ResNet, if all sublayer outputs are zero, the network output equals the input (y = x). This property is important for gradient propagation and initialization stability. In HC, if alpha_l is not 1, this property breaks. How can we keep the benefits of dynamic residual weighting while restoring the identity mapping?

Method

The core of mHC (Manifold-Constrained Hyper-Connections) is to project HC's residual space onto a specific manifold, forcibly restoring the identity mapping property.

1. Define a manifold — a set of parameter configurations that satisfy the identity mapping property. 2. HC's parameter updates are constrained to this manifold. 3. This preserves dynamic adjustment capability while guaranteeing stability during large-scale training.

mHC also includes rigorous infrastructure optimization (memory access optimization, communication optimization) to keep the efficiency overhead of dynamic connections under control.

Key claims

"effective for training at scale"
"tangible performance improvements and superior scalability"
Solves HC's "severe training instability and restricted scalability"

Assessment

mHC is HC's "theoretical completion". It shows that dynamic residual weights must not only "work" but also be "mathematically correct". The manifold constraint guarantees training stability in deep networks (100+ layers). DeepSeek's parallel "engineering + theory" approach on in-house architectures is cited as one reason for rapid iteration on Chinese large models.

Feynman-style commentary (from the original post)

> mHC's mindset is "break first, then repair". HC first breaks the identity mapping (for flexibility); mHC restores it (for stability). This break-and-repair cycle is the standard pattern of deep architecture evolution: ResNet broke "direct mapping" → adding residuals restored trainability; HC broke "fixed weights" → mHC restored stability via a manifold constraint. A good physicist is not afraid of breaking things — as long as you can fix them, and fix them better. Destruction is the beginning of discovery; repair is the completion of understanding.

---

Original tags: 论文深度研究, 小凯, 残差连接