Predicting Where Readers Stumble with "Energy": Hopfield Networks Return to Computational Psycholinguistics

A New Answer to an Old Question

Have you ever been reading a sentence smoothly and suddenly gotten stuck on one word?

For example: "The reporter that the senator who the lobbyist attacked criticized resigned yesterday."

You probably pause at "resigned"—you need to untangle who criticized whom, who attacked whom, and who resigned. This difficulty is called nested clause processing difficulty, a classic phenomenon psycholinguistics has studied for forty years.

Predicting where readers "stumble" has long been a core problem in computational psycholinguistics. For the past decade, two metrics have dominated: surprisal (how unexpected the next word is to the model) and attention entropy (how dispersed the model's attention is). Each is effective but captures a different dimension.

Jakub Dotlačil and Ece Takmaz of Utrecht University propose a third candidate: energy—the energy value from an Energy-Based Transformer. They find this single metric may simultaneously contain the information of both surprisal and attention entropy—a "unified predictor" of reading difficulty.

What Is an "Energy-Based Transformer"?

Ordinary Transformers (the GPT family) work serially: input passes through layer 1, its output goes to layer 2, and so on, each layer doing attention-then-feedforward in sequence.

The Energy-Based Transformer (called NRGPT in the paper) differs in two key ways:

1. Parallel structure

NRGPT computes attention and feedforward in parallel: the outputs of both are added simultaneously to the input at the same layer.

2. Iterative recurrence

This is the crucial difference: NRGPT reuses the same transformer block. A standard GPT with 12 layers has 12 distinct layers; NRGPT may have a single block iterated 12 times.

What does that sound like? A Hopfield network.

The Hopfield network, proposed in 1982, is a classic associative memory model: starting from an initial state, the network iteratively updates until it converges to a stable low-energy state (an attractor). Its energy function defines an energy landscape, and the network rolls downhill like a ball to the nearest minimum.

NRGPT's iteration does essentially the same: each iteration lowers energy until converging to a low-energy configuration. The paper's Figure 1 shows energy decreasing as iterations proceed, with predicted next tokens becoming increasingly plausible.

Moreover, NRGPT's inference is gradient descent: each iteration step updates along the gradient of the energy function. This makes "energy" not just a metaphor but a computable mathematical quantity.

Why Can "Energy" Predict Reading Difficulty?

The authors offer two theoretical reasons:

Reason 1: Energy correlates with transition probability

In energy-based models, the transition probability between two states depends on their energy difference: larger differences mean lower probability. This matches the surprisal intuition—"unexpected" means a low-probability transition, which corresponds to a large energy gap.

Reason 2: Energy correlates with attention entropy

When attention is spread across many tokens (high attention entropy), the model "hesitates" about where to look. That hesitation shows up in the energy: it decreases more slowly, because the model hasn't found a clear downhill direction.

If both hold, energy may encode the information of both surprisal and attention entropy—one metric doing the work of two.

Empirical Validation on Three Corpora

The paper tested energy's predictive power on three standard reading-time corpora:

1. Natural Stories: naturalistic narrative texts with reading-time data 2. UCL Eye-Tracking: fixation times as a proxy for reading difficulty 3. UCL Self-Paced Reading: readers press a key to advance

Result: in all three corpora, energy is a robust predictor of reading times and provides significant additional fit beyond surprisal.

In other words, after predicting reading times with surprisal, adding energy still improves predictions—energy captures information surprisal does not cover.

Single-Layer Energy Captures the Object/Subject Asymmetry

The paper also ran a controlled experiment on relative clause processing, a classic psycholinguistic paradigm—object relative clauses ("The reporter that the senator attacked…") are harder than subject relatives ("The reporter that attacked the senator…").

Finding: energy captures this classic asymmetry at a single layer (iteration step). Object relatives show higher energy than subject relatives, with the difference most pronounced at specific iteration steps.

More importantly, the paper finds energy subsumes the effects of surprisal and attention entropy. When all three metrics enter a regression together, energy remains significant while the independent contributions of surprisal and attention entropy are substantially weakened.

This suggests energy may be a "unified predictor"—the reading-difficulty dimensions that once required two complementary metrics might be covered by one.

Why This Matters

1. A new tool for computational psycholinguistics

Since surprisal entered psycholinguistics in the 2010s, the field has used "next-token probability" as a proxy for reading difficulty. But surprisal captures only prediction difficulty, not processing dynamics. Energy offers a richer signal—not just "how surprised the model is" but "how much the model is still struggling to converge."

2. A Hopfield renaissance

Hopfield networks starred in connectionism in the 1980s–90s but were sidelined by deep learning. Their recent return to modern AI (Modern Hopfield Networks, Dense Associative Memory) has renewed interest. This paper links energy-based models to human reading behavior, giving the Hopfield framework a new application.

3. "Unified vs. complementary" philosophy of science

For a decade, the consensus was that multiple complementary metrics are needed. This paper challenges that: perhaps we simply hadn't found the unified metric yet. Energy may be it—echoing (cautiously) the physicist's dream of unification.

An Honest Assessment

Strengths

• Clear theoretical motivation: derived from the Hopfield framework rather than trial-and-error
• Cross-corpus validation: not an overfit to a single dataset
• Both controlled experiments and corpus data
• "Subsumes" analysis: examines whether energy contains other metrics' information, not just whether it works

Limitations

• Only one energy-based model (NRGPT) was tested
• Energy-based models remain small: NRGPT's scale is far below modern LLMs
• The "subsumes" statistical claim needs caution: with highly correlated predictors, regression coefficient estimates are unstable; multicollinearity may partly drive the conclusion
• No comparison with more modern metrics such as LLM hidden states or probe-based measures
• Causal direction unclear: does high energy cause slow reading, or do complex sentences cause both? This is a correlational study

A Provocative Extension

The most exciting part of this paper is not the specific results but a philosophical question: if a Transformer's energy function can predict human reading difficulty, does the Transformer "understand" language in some mathematical sense?

Not in the Turing-test sense—that bar is too low—but in the sense of "isomorphism of processing": the model's struggle to converge correlates highly with the human struggle to comprehend.

This correlation may be a coincidence—both driven by syntactic complexity. But it may point to something deeper: human brains and Transformers may share an underlying computational geometry when processing language. Hopfield networks connect naturally to neural dynamical systems; if a Transformer's energy landscape resembles the brain's neural dynamics landscape, the energy–reading-time correlation is not coincidence but isomorphism.

That is speculation—the paper doesn't explore it. But good papers don't just answer questions; they open new ones. This paper does.

---

Paper: https://arxiv.org/abs/2606.23382 HTML version: https://arxiv.org/html/2606.23382v1 Code: jakdot/energy-transformers-reading-difficulty Authors: Jakub Dotlačil, Ece Takmaz (Utrecht University)