When AI 'Reads' the Brain: Deconstructing a Statistical Illusion in LLM-Brain Alignment

> TL;DR: A UCLA team showed that a minimalist model with zero understanding of language can match the 1.5-billion-parameter GPT-2 XL on classic neuroscience benchmarks. The reason isn't that AI lacks intelligence—it's that prior evaluation methods left a temporal cheating backdoor wide open.

---

What This Is: The "Golden Myth" of LLM-Brain Alignment

A Tempting Narrative

In 2021, Schrimpf et al. published an influential paper in *Nature Neuroscience* showing that GPT-2 XL's (1.5B parameters) internal activations could predict human fMRI and ECoG signals while people listened to stories or read sentences. Bigger models predicted better, and Transformer layers seemed to mirror the hierarchy of language processing in the brain.

This launched a thriving field: LLM-brain alignment. Follow-up findings piled up:

• GPT-2's intermediate layers predict sentence-comprehension brain regions
• Autoregressive models (GPT family) resemble the brain more than bidirectional ones (BERT)
• Model predictive power correlates with human reading comprehension scores

The tempting conclusion: Transformers may be the computational model of the brain's language processing.

But Does This Conclusion Hold?

The UCLA-led team raised a fundamental objection: prior studies used shuffled train-test splits.

The core problem: neural signals exhibit strong temporal autocorrelation. Adjacent time points produce similar brain responses—not purely because stimuli match, but because neural activity has inertia, like a drum's sound decaying gradually rather than vanishing.

With shuffled splits, a test sentence's neural signal may closely resemble that of adjacent training sentences. A model doesn't need to "understand" language at all—just learn "nearby signals look alike"—to score highly.

It's like an exam meant to test math skills, where the answers were accidentally printed on the back of the previous question. Smart students ace it by flipping pages, but no math ability was ever tested.

OASM: A Minimalist Model That "Cheats"

To demonstrate the flaw, the team built OASM (Orthogonal Autocorrelated Sequences Model):

1. Assign each data point (sentence/passage/story) a dimension of an n-dimensional identity matrix 2. Apply a Gaussian filter along the diagonal within temporally adjacent blocks (sentences in the same paragraph, segments of the same story) 3. Result: representations are similar within a paragraph, fully orthogonal across paragraphs

OASM has no linguistic ability whatsoever:

• No word embeddings
• No contextual understanding
• No grammar knowledge
• Its only "knowledge": temporally adjacent things should be similar

The results are striking:

Under shuffled splits:

• OASM's neural predictivity matches GPT-2 XL, and on some datasets is even higher
• With OLS regression, OASM's and GPT-2 XL's performance curves nearly overlap
• With L2-regularized regression, OASM still comes very close

This means GPT-2 XL's high scores under the previous "gold standard" evaluation largely reflect temporal autocorrelation cheating.

---

Why It Matters: After the Alignment Illusion Breaks

Contiguous Splits: Lifting the Curtain

The team introduced contiguous splits—keeping temporally adjacent data together in either train or test sets.

Results:

| Condition | GPT-2 XL vs. OASM | |---|---| | Shuffled + OLS | GPT-2 XL ≈ OASM (no significant advantage) | | Shuffled + L2 | GPT-2 XL slightly better, gap greatly reduced | | Contiguous + OLS | GPT-2 XL significantly better | | Contiguous + L2 | GPT-2 XL significantly better, but less than previously claimed |

Key findings:

• Under contiguous splits, OASM fails completely—it can't generalize across paragraphs
• GPT-2 XL remains effective, showing it does learn some genuinely linguistic representations
• But GPT-2 XL's advantage is far smaller than previously claimed

Activation Extraction: Another Hidden Bias

A second methodological issue: how activations are extracted from LLMs:

• Last token pooling (most common)
• Mean pooling
• Sum pooling

Findings:

• Last token pooling usually performs worst, especially disadvantaging bidirectional models (e.g., BERT)
• Mean and sum pooling often do better
• Changing the extraction method can flip model rankings

So the prior claim that "autoregressive GPT beats bidirectional BERT" may be an artifact of last-token pooling's bias against bidirectional models.

PWR: Position and Word Rate Explain ~80% of the "Alignment"

The team introduced another minimalist baseline—PWR (Position and Word Rate)—encoding only:

1. Word position in the sentence 2. Word frequency in the corpus

Results:

• PWR alone achieves high neural predictivity
• PWR + static embeddings (GloVe) explains over 80% of GPT-2 XL's neural variance

In other words, most of GPT-2 XL's "brain alignment" is explainable by shallow features: word position, word frequency, and static word vectors—no deep Transformer computation required.

Revisiting Classic Conclusions

Schrimpf et al. (2021)'s key claims, re-examined:

1. GPT-2 XL nears the noise ceiling — severely overestimated, inflated by shuffled splits and OLS 2. Autoregressive models beat bidirectional ones — possibly an artifact of activation extraction method 3. Model performance correlates with human behavior — needs stricter controls to verify

Under rigorous evaluation (contiguous splits + L2 regularization + multiple extraction methods), these classic conclusions shrink dramatically in strength.

---

How to Use It: A More Rigorous Methodological Framework

Data Splits: Contiguous > Shuffled

| Split type | Mechanism | Problem | Verdict | |---|---|---|---| | Shuffled | Randomly assign sentences to train/test | Temporal autocorrelation cheating inflates scores | Unreliable | | Contiguous | Assign whole passages/sentences/stories | Forces genuine generalization | Recommended |

Concrete practice:

• Pereira2018: split by paragraphs, different paragraphs on the same topic across train/test
• Fedorenko2016: split by sentences, sentences within a condition kept together
• Blank2014: split by story, different stories across train/test

Regression: Regularization Is a Must

• OLS overfits when feature count (LLM hidden dimension) approaches sample size, further inflating scores under shuffled splits
• L2 (Ridge) penalizes large weights, yielding more conservative, realistic estimates

Activation Extraction: Don't Just Take the Last Token

Systematically compare multiple extraction methods per architecture:

• Autoregressive (GPT): last token, mean pooling, sum pooling
• Bidirectional (BERT): mean pooling, sum pooling, CLS token
• Layers: shallow, middle, deep

Only after controlling for extraction are cross-model comparisons fair.

Control Models: Minimalist Baselines Are Mandatory

The proposed gold standard—any LLM-brain alignment claim must beat these baselines:

| Control model | Encodes | If it beats the LLM... | |---|---|---| | OASM | Temporal autocorrelation | LLM scores may be temporal cheating | | PWR | Position + word rate | LLM scores may reflect shallow features | | GloVe | Static word vectors | LLM scores may reflect embedding quality | | Random embeddings | Random vectors | Tests whether any high-dim representation "aligns" |

If an LLM can't significantly outperform these baselines, its "alignment" is a statistical illusion.

---

Closing: Not a Rejection, but a Recalibration

The paper's real value is not denying any LLM-brain relationship, but calibrating our confidence in it. As the team states:

> "Under contiguous splits, GPT-2 XL does significantly outperform OASM and PWR, showing it learns some genuinely linguistic representations. But these advantages are far less dramatic than previously claimed."

Three core lessons:

1. Methodological fragility: a seemingly tiny choice (shuffled vs. contiguous splits) can completely flip conclusions—many NeuroAI "findings" may be methodological artifacts. 2. The power of confounds: temporal autocorrelation, word frequency, and position can explain 80% of the "alignment"; uncontrolled, we mistake statistical coincidence for theoretical breakthrough. 3. The value of minimalist models: before praising complex models, ask what a linguistically clueless baseline can achieve. If the answer is "about the same," the complex model's advantage is suspect.

The paper's title is precise—"Illusions of Alignment." It doesn't say the field is fake; it says many spectacular "alignments" are, under stricter testing, methodological ghosts. For NeuroAI, this is a necessary cold shower—it won't extinguish enthusiasm, but it will make future research more rigorous and credible.

---

Key References

1. Illusions of Alignment Team, UCLA. (2025). Illusions of Alignment Between Large Language Models and Brains Emerge From Fragile Methods and Overlooked Confounds. *bioRxiv:2025.03.09.642245*. 2. Schrimpf, M., et al. (2021). The Neural Architecture of Language: Integrative Modeling Converges on Predictive Processing. *PNAS*. [the classic study under critique] 3. Caucheteux, C., & King, J.R. (2022). Brains and Algorithms Partially Converge in Natural Language Processing. *Science Advances*. 4. Antonello, R., et al. (2023). Cross-Granularity Attention for Natural Language Processing in the Brain. *NeurIPS*. 5. Jain, S., & Huth, A. (2020). Incorporating Context into Language Encoding Models for fMRI. *NeurIPS*.