Attention Residuals: Kimi Team Replaces Fixed Residual Connections with Attention over Layers

Forum topic · ✨步子哥 · 2026-03-17

Summary

The Kimi Team's new paper, Attention Residuals (AttnRes), proposes replacing the standard PreNorm fixed-weight residual connection in large language models with a softmax attention mechanism over all previous layers' outputs. This makes each layer's contribution input-dependent and learnable, resolving the uncontrolled growth of hidden states and the uniform dilution of per-layer contributions that plague conventional residuals, while yielding more uniform output magnitudes and better-balanced gradients. To control the memory and communication overhead of full attention over all prior layers, the authors introduce Block AttnRes, which applies attention at the block level combined with cached pipeline communication and a two-stage computation strategy, keeping costs close to ordinary residuals. Experiments show consistent scaling-law gains, ablations confirm content-dependent depth selection helps, and the method was integrated into the Kimi Linear architecture (48B total parameters, 3B activated), pretrained on 1.4 trillion tokens, improving all downstream tasks.

The Kimi Team (including Guangyu Chen, Yu Zhang, Jianlin Su, Weixin Xu, Siyuan Pan, and 34 authors in total) released a new paper on March 16: Attention Residuals.

arXiv link: https://arxiv.org/abs/2603.15031

The Core Problem

Modern LLMs universally use PreNorm + fixed-weight residual connections. Every layer's output is added with weight 1 to the next layer, which causes:

Hidden states to grow uncontrollably with depth
Each layer's contribution to be uniformly diluted
Uneven gradient distribution, with deep-layer information drowned out by shallow layers

The Idea: Attention Residuals (AttnRes)

The authors propose replacing the fixed residual with softmax attention:

Instead of directly adding the previous layer, each layer performs attention over the outputs of all preceding layers.
Weights become input-dependent and learnable, letting the model decide which layer's representation it needs at each moment.

Effects:

Thoroughly solves the dilution problem of PreNorm
More uniform output magnitudes per layer and more balanced gradient distribution
Significantly improved scalability with depth

Practical Optimization: Block AttnRes

Full attention over all preceding layers brings large memory and communication overhead (especially with MoE / pipeline parallelism). The authors propose Block AttnRes:

Divide layers into several blocks
Perform attention only at the block level
Combine cached pipeline communication + a two-stage computation strategy

→ Overhead is nearly the same as ordinary residuals while retaining most of the benefits, making it practical for large-scale training.

Experimental Validation

Scaling law experiments: consistent performance gains across model sizes
Ablations: confirm that content-dependent depth selection is indeed the source of the gains
Real-world deployment: integrated into the Kimi Linear architecture (48B total parameters, 3B activated), pretrained on 1.4 trillion tokens
Successfully mitigates PreNorm dilution
Improvements on all downstream tasks

TL;DR

The paper upgrades the residual connection from a rigid addition into an intelligent attention-based selection, offering an elegant and practical new approach to scaling Transformer depth. The Kimi Team validated it not just in theory but on a real 48B pretrained model — very high-quality work.