Emergent Structure in Attention: 60 LLMs Show Striking Agreement on One Dimension

> Paper: Emergent Structure in the Marginal Attention Space of Language Models > arXiv: https://arxiv.org/abs/2610.03109 > Code: https://github.com/Flegyas/marginal-attention

A Simple Observation

If you train 60 different language models — different labs, different data, different parameter counts, different tokenizers, different architectures — would their attention matrices look the same?

Intuition says no. The weights differ entirely; how could attention match?

But in October 2026, Valentino Maiorca, Walter Nelson, and Francesco Locatello (Bocconi University / IST Austria) found a counterintuitive result: if you "compress" the attention matrix along the right dimension, attention patterns across models are strikingly similar — but only under one specific compression method.

This reveals that attention matrices hide two completely different structures: one belonging to the text, one belonging to the model.

What Is "Marginal Attention"

First, the terminology. Transformer attention is multi-head; each head at each output position has an attention distribution (a softmax weight vector whose length equals the input length). This is a 3D tensor: [output position, head, input position].

The paper's insight: average this 3D tensor along the output-position axis to get a 2D matrix: [head, input position]. This is "marginal attention."

Why average over output positions? Output positions are the "queries" — they determine "where am I looking now." But the paper is interested not in any specific query's attention, but in "overall, how much each head attends to each input token." It's like moving from "attention at each moment" to "average attention over a text" — the latter is more stable and more informative.

Compression 1: Sum along the head axis → per-token marginal attention

The result is a vector of input length; each element is the total attention all heads give a token.

Key finding: this vector is highly similar across models. Feed the same text to 60+ different LLMs and their total per-token attention is nearly identical.

This means: this vector is a property of the text, not the model. It reflects how important a token is within the passage — an intrinsic, model-independent signal.

Compression 2: Sum along the input-position axis → per-head marginal attention

The result is a vector of head-count length; each element is the total attention a head gives all input tokens.

Key finding: this vector differs greatly across models, but is highly stable across documents within the same model.

This means: this vector is a property of the model, not the text. It reflects a head's "working style" — some heads inherently attend a lot, others little, determined by training rather than input.

Why This Dichotomy Exists

The paper offers a theoretical explanation linking marginal attention to the model's input-output Jacobian.

The Jacobian \(\frac{\partial \text{output}}{\partial \text{input}}\) describes how much each input token influences each output logit — a local description of the model's entire computation, including attention, value matrices, feedforward layers, and a first-order approximation of all nonlinearities.

The paper finds: marginal attention (a statistic of the attention matrix alone) closely approximates the Jacobian's blockwise norms.

This seems strange — attention is only part of the Jacobian; how can it approximate the whole?

The theoretical explanation: for two smooth models, if their next-token distributions are close, their input-output Jacobians are also close. This is a theorem (Theorem 1 in the paper).

Intuition: if two models largely agree on "what comes next," then "how each input token influences that judgment" also agrees — because the latter is the gradient of the former. Since different models produce similar next-token distributions on the same text (they all predict the same language), their marginal attention is similar too. That explains why "marginal attention compressed along the head axis" is a text property — it reflects the text's predictability structure, shared by all models that know the language.

Conversely, compression along the token axis reflects each head's working style — set by training details (initialization, data order, regularization), so it differs across models but stays stable within one model across documents (head "personality" is fixed during training).

Practical Application: KV Cache Eviction

Beyond theory, the paper gives a practical application: KV cache eviction via marginal attention.

The KV cache dominates Transformer inference memory for long contexts. Eviction keeps "important" tokens and drops "unimportant" ones to save space.

The question: how to decide each head's cache budget? Traditional approaches use one budget for all heads (too coarse) or recompute per document (too slow).

The paper's approach: 1. Offline: compute each head's marginal attention on pretraining-style text 2. Set each head's KV cache budget from this "head personality" — high-attention heads get more, low-attention heads less 3. Online during inference: use the precomputed budget plus a simple training-free token score (based on per-token marginal attention)

Result: on standard eviction benchmarks, this matches methods that recompute budgets per document or train an eviction model — but budgets are precomputed offline, with zero additional inference overhead.

This directly exploits the dichotomy:

• Per-head marginal attention (model property) → offline budget precomputation
• Per-token marginal attention (text property) → online token-importance scoring

The Data

• 60+ LLMs evaluated, spanning labs, parameter counts (2B to 70B+), tokenizers, architectures
• Per-token marginal attention cross-model correlation: highly consistent (Figure 1)
• Per-head marginal attention within-model cross-document correlation: highly stable
• Jacobian approximation: correlation between marginal attention and Jacobian blockwise norms > 0.9
• KV cache eviction: comparable to SOTA on standard benchmarks, with zero online overhead

Why It Matters

1. "Representation convergence" extends from embeddings to attention

Prior work found different models' embedding spaces are similar (Kornblith 2019, Raghu 2017, Morcos 2018, Moschella 2023). This paper extends that from embeddings to attention — not only are final representations converging, the intermediate attention mechanism converges too.

This suggests an "invariant of attention": once a model learns the same language, its attention patterns converge to the same structure. Nobody designed this structure — it is emergent.

2. A "text property vs. model property" dichotomy framework

The paper's deepest contribution is a binary framework: for any internal model statistic, ask "is this a property of the text or of the model?"

• Text property: consistent across models on the same text → enables cross-model transfer
• Model property: consistent across documents within one model → enables cross-document precomputation

The framework extends to other statistics:

• Feedforward activation patterns: text or model property?
• Residual stream norms: text or model property?
• Per-layer "information flow": text or model property?

Each can be tested the same way: compress along different axes and see which axis shows cross-model consistency and which shows cross-document consistency.

3. Theoretical value of the Jacobian connection

The finding that marginal attention approximates Jacobian blockwise norms links a simple attention statistic to a description of the model's entire computation.

This means: the attention matrix is not just a record of "where the model looks" — it implicitly encodes "how much inputs affect outputs", normally requiring backpropagation. But marginal attention only needs forward-pass attention weights — a nearly free statistic.

This may open a new research direction: approximating backward-pass quantities with forward-pass statistics. If attention can approximate the Jacobian, other forward statistics might approximate influence functions, gradient attributions, and more — drastically cutting interpretability research costs.

4. Practical value: zero-overhead KV cache eviction

The KV cache is the main bottleneck of long-context inference. Existing methods lack precision (one budget for all heads) or speed (per-document recomputation). Marginal attention offers a third path: offline per-head budgets, zero online cost.

Deployable immediately — no architecture changes, no training, one pass of marginal-attention statistics over pretraining text. For inference providers (OpenAI, Anthropic, local deployments), it's a nearly free optimization.

Honest Limitations

1. Smoothness assumption. The theory assumes models are smooth. ReLU networks aren't strictly smooth, yet the approximation holds empirically — not fully explained. 2. First order only. The Jacobian is a first-order approximation; high-order effects are ignored. The paper gives empirical estimates for "how much Jacobian information attention captures" but no theoretical bound. 3. KV eviction isn't SOTA. The method is "comparable" to recomputation but doesn't significantly exceed it. Marginal attention's practical value may lie more in diagnosis and understanding than performance gains. 4. Model scope. The 60+ LLMs are broad but all decoder-only Transformers. Whether encoder-decoder, MoE, or non-Transformer architectures show the same phenomenon is unknown.

A Bigger Picture

This paper suggests a deeper truth: complex systems often hide simple invariants inside.

Sixty differently trained models — different data, architectures, parameter counts spanning 30x — have nearly identical attention matrices under one specific compression. This means: however you train, once you learn the same language, your attention converges to the same structure.

This structure wasn't designed — no one told the models how to attend. It is emergent, an inevitable result of learning the same language.

It resembles "universality classes" in physics — systems with different microscopic mechanisms showing identical macroscopic behavior. In statistical mechanics, different lattice models near criticality share one scaling function. In language models, differently trained models share one function describing their marginal attention.

Perhaps the statistical structure of language itself is an attractor: wherever you start, if you predict the same language, you are drawn to the same attention patterns. If true, this matters more than any specific KV-cache optimization — it means "representation convergence" in language models is not an accident but a mathematical property of language itself.

---

Paper: https://arxiv.org/abs/2610.03109 Code: https://github.com/Flegyas/marginal-attention