Attention Residuals: Kimi Team Replaces Fixed Residual Connections with Cross-Layer Attention

Forum topic · ✨步子哥 · 2026-03-17

Summary

The Kimi Team (Moonshot AI) proposes Attention Residuals (AttnRes), a new architecture that replaces the standard fixed-weight residual connection in PreNorm Transformers with a softmax attention over the outputs of all preceding layers. This makes each layer's contribution input-dependent and learnable, addressing three known problems of fixed residuals: uncontrolled growth of hidden-state magnitudes with depth, uniform dilution of per-layer contributions, and imbalanced gradient flow. To make the approach practical at scale, the authors introduce Block AttnRes, which partitions layers into blocks and performs attention only at the block level, combined with cached pipeline communication and two-stage computation, so that overhead is close to that of ordinary residuals. Experiments show consistent gains across model scales in scaling-law studies, ablations confirm that content-dependent depth selection drives the improvement, and the method was integrated into the Kimi Linear architecture (48B total parameters, 3B activated) and pretrained on 1.4 trillion tokens, mitigating PreNorm dilution and improving all downstream tasks. Paper: https://arxiv.org/abs/2603.15031

The Kimi Team (including Guangyu Chen, Yu Zhang, Jianlin Su, Weixin Xu, Siyuan Pan, among 34 authors in total) released a new paper on March 16, 2026: Attention Residuals.

The Core Problem

Modern LLMs universally use PreNorm + fixed-weight residual connections. Each layer's output is added with weight 1 to the next layer, which causes:

Hidden states to grow without control as depth increases
Each layer's contribution to be uniformly diluted
Imbalanced gradients, with deep-layer information drowned out by shallow layers

The Key Idea: Attention Residuals (AttnRes)

The authors propose replacing the fixed residual with softmax attention:

Instead of directly adding the previous layer's output, each layer performs attention over the outputs of all preceding layers.
The weights become input-dependent and learnable, letting the model decide "which layer's representation it needs most right now."

Benefits:

Completely solves the PreNorm dilution problem
More uniform output magnitudes per layer and more balanced gradient distribution
Significantly improved scalability with depth

Practical Optimization: Block AttnRes

Full attention over all preceding layers brings large memory and communication overhead, especially under MoE / pipeline parallelism. The authors propose Block AttnRes:

Layers are divided into blocks
Attention is performed only at the block level
Combined with a cached pipeline communication + two-stage computation strategy

This keeps overhead nearly identical to ordinary residuals while retaining most of the benefits — genuinely deployable for large-scale training.

Experimental Validation

Scaling-law experiments: consistent improvements across model sizes
Ablation studies: confirm that "content-dependent depth selection" is what drives the gains
Real-world deployment: integrated into the Kimi Linear architecture (48B total parameters, 3B activated) and pretrained on 1.4 trillion tokens
Successfully mitigates PreNorm dilution
Improvements across all downstream tasks

TL;DR

The paper upgrades the residual connection from a rigid addition to an intelligent, attention-based selection, offering an elegant and practical new approach to scaling Transformer depth. The Kimi team validated it all the way up to a 48B pretrained model — high-quality work.