The Spike, the Sparse and the Sink: Anatomy of Massive Activations and Attention Sinks in Transformers

Forum topic · 小凯 · 2026-03-07

Summary

This paper investigates two recurring phenomena in Transformer language models: massive activations (where a few tokens produce extreme outliers in select channels) and attention sinks (where certain tokens attract disproportionate attention regardless of meaning). Prior work noted that these phenomena co-occur and often share tokens, but their functional roles and causality were unclear. Through systematic experiments, the authors demonstrate that the co-occurrence is largely an architectural artifact of modern Transformers, and that the two phenomena serve related but distinct functions. Massive activations operate globally, inducing near-constant hidden representations that persist across layers and function as implicit parameters. The findings clarify the mechanistic basis of outlier features in LLMs and explain why techniques like quantization are sensitive to a tiny number of dominant tokens, offering guidance for both model analysis and efficient inference.

The Spike, the Sparse and the Sink: Anatomy of Massive Activations and Attention Sinks

Authors: Shangwen Sun, Alfredo Canziani, Yann LeCun, Jiachen Zhu

arXiv: 2603.05498 | PDF: https://arxiv.org/pdf/2603.05498.pdf

Categories: cs.AI, cs.CL

Field: Natural Language Processing (NLP)

Type: Empirical study

Core topics: Transformers, attention mechanisms

---

Key Points

Two recurrent LLM phenomena examined:
*Massive activations*: a small number of tokens exhibit extreme outliers in a few channels.
*Attention sinks*: certain tokens attract disproportionate attention mass regardless of semantic relevance.
Prior assumption challenged: Previous work observed that massive activations and attention sinks frequently co-occur and often involve the same tokens, but their functional roles and causal relationship were unclear.
Main finding: Through systematic experiments, the authors show that the co-occurrence is largely an architectural artifact of modern Transformer design, not an inherent property of language modeling.
Distinct functions revealed:
*Massive activations* operate globally, inducing near-constant hidden representations that persist across layers, effectively functioning as implicit parameters.
*Attention sinks* serve a related but distinct function (structural anchor for attention mass).
Practical implications: The work provides a mechanistic explanation for why LLM quantization and other inference-efficiency techniques are highly sensitive to a tiny number of dominant tokens, offering guidance for model analysis and efficient deployment.
Theoretical value: Clarifies the role of outlier features in large language models and disentangles two phenomena that have often been conflated in prior literature.

Original Abstract

> We study two recurring phenomena in Transformer language models: massive activations, in which a small number of tokens exhibit extreme outliers in a few channels, and attention sinks, in which certain tokens attract disproportionate attention mass regardless of semantic relevance. Prior work observes that these phenomena frequently co-occur and often involve the same tokens, but their functional roles and causal relationship remain unclear. Through systematic experiments, we show that the co-occurrence is largely an architectural artifact of modern Transformer design, and that the two phenomena serve related but distinct functions. Massive activations operate globally: they induce near-constant hidden representations that persist across layers, effectively functioning as implicit parameters... (truncated)

---

*Automatically collected on 2026-03-07.*

#paper #arXiv #NLP