Brain-AI Alignment Illusion: Human-Like Attention Heads Aren't the Ones Doing the Work

An Experiment That Makes Neuroscientists Uneasy

Imagine observing the internals of an LLM—thousands of attention heads working in parallel, each distributing attention across the sequence. Now imagine you also have EEG data from humans performing the same task.

You would naturally ask a seemingly obvious question: which attention heads in the model have activity patterns most like the human brain?

And a more critical follow-up: are these "most brain-like" heads also the ones the model actually relies on?

Intuition says "they should be." This paper shows: they are not.

And the gap is not small—it's orders of magnitude off.

Experimental Design: AAABAAA → B

The research team used a strikingly elegant task: abstract sequence completion. The model sees a symbol string like AAABAAA and must infer the next symbol is B. Eight abstract patterns, three presentation formats (English words, German words, abstract symbols), and two response modes (open-ended, multiple choice).

Human participants performed the same task while EEG (brain-wave) and eye-tracking data were recorded. The team then computed three scores:

1. Brain Score: correlation between each head's representational dissimilarity matrix (RDM) and the RDM of human frontal FRPs (fixation-related potentials). Higher scores mean the head's "geometry of thought" resembles the human brain more.

2. Patching Score: a direct measure of causal importance via activation patching. Transplant a head's output from a clean prompt into a corrupted prompt and measure how much the probability of the correct answer improves.

3. Concept Score: whether the head encodes the abstract pattern itself (a representation invariant across formats and languages).

Three scores, three dimensions, same set of heads—then compare their relationships.

Core Finding: Alignment ≠ Causality

Finding 1: Brain alignment and causal importance are nearly unrelated

In Llama-3.1-70B, the Spearman correlation between brain scores and causal scores is only ρ = 0.03. Across a replication cohort of 17 models, the mean is ρ = 0.017 (range [-0.017, 0.077]).

In plain terms: whether a head "looks like the human brain" has essentially no relationship to whether it matters.

Finding 2: Deleting brain-aligned heads barely hurts performance

Deleting heads ranked by different scores:

• Causally-ranked (FV-ranked): removing 12.5% of heads drops accuracy by 43 percentage points
• Concept-ranked (CV-ranked): peak drop of 20.7 points
• Brain-ranked: peak drop of only 12.8 points, and it takes removing 24.5% of heads to get there

Brain-ranked ablation is less than a third as damaging as causal ablation, while requiring twice as many deletions.

Finding 3: Brain-aligned heads form two families—repetition and novelty

Clustering the top 20 brain-aligned heads reveals two distinct attention patterns:

• Repetition heads: attend to repeated elements (the A's in AAABAAA)
• Novelty heads: attend to the unique element (the lone B)

Both families replicate independently across all 17 models—not a quirk of one model.

Finding 4: Novelty heads are "human-like but not doing the work"

The most counterintuitive result. Novelty heads' attention patterns match human eye movements closely (positively correlated in all 17 models, mean ρ = 0.373). Humans stare at the B; novelty heads also stare at the B.

But delete them? Accuracy barely changes. After ablating 800 heads, accuracy falls from 71.5% to 63.5%—while the random-deletion control is already below 50%. Novelty-head deletion is even *less* harmful than random deletion.

Across 8 additional tasks (few-shot, prose, retrieval, coreference, etc.), novelty-head deletion was less harmful than random deletion on 7 of them.

The model is "looking at" the novel element, but not "using" it.

Finding 5: Repetition heads help, but aren't the core

Repetition-head deletion is more harmful than random deletion (peak excess of 11.2 points), but far from causal heads. Intriguingly, after deleting 130 repetition heads (a 5.5-point accuracy drop), concept representations remain fully intact—concept heads in later layers still classify the abstract patterns with 100% accuracy. Repetition heads co-occur with concept representations (median layers 28 vs 33) but are not their source.

Why? Softmax's "Attention Illusion"

The paper offers an elegant explanation: novelty heads may be spandrels—a concept from evolutionary biology. Spandrels in architecture (like those at St. Mark's Basilica in Venice) are structural byproducts of arches meeting domed ceilings, later decorated with mosaics but not designed for decoration.

Softmax has a property: every head must distribute attention across all tokens, even if its output barely affects the answer. So a head can display very "meaningful" attention patterns—like staring at the novel element—while that pattern is merely a byproduct of architecture and training, not functional computation.

Humans likewise fixate on novel elements either because it's "the answer" (task-driven) or because it "stands out" (attentional capture). The paper's data cannot distinguish these—but the ablations show that even an attention pattern identical to human gaze is not necessary for the model to complete the task.

Validation Across 17 Models

All findings replicate across:

• Llama-3.1-70B/8B (base + instruct)
• Qwen2.5 series (3B–72B, base + instruct)
• Phi-4 (14B)
• DeepSeek-R1-Distill-Llama-70B

Weak brain-causal correlation, the repetition/novelty split, and the harmlessness of novelty-head deletion all reproduce across models. The only variation: the brain–concept score relationship turns negative in Qwen instruct models, showing this relationship depends on model family and training.

Implications for AI Interpretability

1. Correlation is not causation. A unit resembling brain signals doesn't mean it performs useful computation.

2. Attention patterns ≠ function. Where a head "looks" and what it "uses" are different things; softmax-forced attention can be functionally inert.

3. Human similarity may be a spandrel. Emergent brain-like features may be byproducts of architectural constraints, not functional adaptations—mirroring Gould & Lewontin's (1979) critique in evolutionary biology.

4. Causal intervention is the gold standard. Only "delete it and see what happens" separates "doing computation" from "appearing to do computation." RSA and brain scores are not enough; activation patching is.

A Cross-Domain Analogy: When "Looks Like" Isn't "Is"

Convergent evolution produces structures that look similar but differ in function and developmental path—bird and insect wings, dolphin and shark body shapes. Brain-AI alignment may be analogous: models and brains may "look alike" in representational space while using entirely different computational paths. Human-like attention allocation does not imply a shared reasoning route.

Personal Reflection

How often in AI research do we take "looks like" for "actually is"?

• Generated text "looks like" reasoning → possibly statistical pattern matching
• Attention weights "look like" focusing on key information → attention ≠ causal contribution
• Brain alignment "looks like" shared computation → alignment ≠ causal importance

Evolutionary biology took nearly a century to systematically distinguish adaptations from spandrels. AI research may need a similar cognitive upgrade. Next time you see a "brain-like" model component, ask yourself: what happens if you delete it?

---

Paper: arXiv:2609.37991 Authors: Pinier, Opiełka, Rosenbusch, Webb, Nunez Key concepts: brain alignment ≠ causal importance, attention spandrels, repetition vs. novelty heads, representational similarity vs. causal intervention