Predicting Where Readers Stumble with "Energy": Hopfield Network Regression in Computational Psycholinguistics

A New Answer to an Old Question

Have you ever read a sentence smoothly, then suddenly gotten "stuck" on a particular word?

For example: "The reporter that the senator who the lobbyist attacked criticized resigned yesterday."

You probably pause at "resigned"—because you need to untangle who criticized whom, who attacked whom, and who resigned. This reading difficulty is called nested relative clause processing difficulty, a classic phenomenon studied in psycholinguistics for forty years.

Predicting where readers stumble has been a core problem in computational psycholinguistics. For the past decade, the field has been dominated by two metrics: surprisal (how surprised the model is by the next word) and attention entropy (how dispersed the model's attention is). Each is effective, but they capture different dimensions—surprisal measures "prediction difficulty," attention entropy measures "attentional dispersion."

Jakub Dotlačil and Ece Takmaz of Utrecht University propose a third candidate: energy—the energy value from an Energy-Based Transformer. They find that this single metric may contain the information of both surprisal and attention entropy, serving as a "unified predictor" of reading difficulty.

What Is an Energy-Based Transformer?

To understand the paper, you first need to know what an Energy-Based Transformer is.

A standard Transformer (the GPT family) works serially: input passes through layer 1, its output goes to layer 2, and so on—each layer is a sequential "attention, then feed-forward" structure.

The Energy-Based Transformer (called NRGPT in the paper) differs in two key ways:

1. Parallel Structure

In a normal Transformer, attention and feed-forward layers are serial (attention first, then feed-forward). NRGPT makes them parallel: at each layer, attention output and feed-forward output are simultaneously added to the input.

2. Recursive Iteration

This is the crucial difference: NRGPT reuses the same transformer block. A standard GPT with 12 layers has 12 distinct layers; NRGPT may have just 1 block, iterated 12 times.

What does that sound like? A Hopfield network.

The Hopfield network, proposed in 1982, is a classic associative memory model: given an initial state, the network updates iteratively until it converges to a stable low-energy state (an attractor). Its "energy function" is central—it defines an energy landscape, and the network rolls downhill like a ball to the nearest minimum.

NRGPT's iteration essentially does the same thing: each iteration lowers the energy until it converges to a low-energy configuration. Figure 1 of the paper shows this—as iterations proceed, energy decreases and the predicted next token becomes increasingly "plausible."

Moreover, NRGPT's inference process is gradient descent: each iterative step updates along the gradient of the energy function. This makes "energy" not just a metaphor but a computable mathematical quantity.

Why Energy Predicts Reading Difficulty

This is the paper's core question, and the authors give two theoretical reasons:

Reason 1: Energy correlates with transition probability

In energy-based models, the transition probability between two states depends on their energy difference: the larger the difference, the lower the probability. This matches the surprisal intuition—"surprise" means a low-probability transition, and low-probability transitions correspond to large energy differences.

Reason 2: Energy correlates with attention entropy

When attention is dispersed across many tokens (high attention entropy), the model is "hesitating" about where to look. This hesitation is reflected in the energy—energy decreases more slowly because the model hasn't found a clear "downhill direction."

If both reasons hold, energy may simultaneously encode the information in surprisal and attention entropy—one metric doing the work of two.

Empirical Validation Across Three Corpora

The paper tested energy's predictive power on three standard reading-time corpora:

1. Natural Stories: naturalistic narrative text with reading-time data 2. UCL Eye-Tracking: eye-tracking corpus, with fixation durations as a proxy for reading difficulty 3. UCL Self-Paced Reading: readers press a key to advance through the text

Result: in all three corpora, energy was a robust predictor of reading times and provided significant additional fit above and beyond surprisal.

In other words, after using surprisal to predict reading time, adding energy further improves prediction—meaning energy captures information surprisal does not cover.

Single-Layer Energy Captures the Object/Subject Asymmetry

The paper also ran a controlled experiment on relative clause processing, a classic psycholinguistic paradigm—object relative clauses ("The reporter that the senator attacked…") are harder to comprehend than subject relative clauses ("The reporter that attacked the senator...").

The finding: energy captures this classic asymmetry at a single layer (iteration step). Object relative clauses show higher energy than subject relative clauses, and the difference is most pronounced at specific iteration steps.

More importantly, the paper finds that energy subsumes the effects of surprisal and attention entropy. When all three metrics are entered into a regression together, energy's effect remains significant while the independent contributions of surprisal and attention entropy are greatly weakened.

This suggests a possibility: energy may be a "unified predictor"—the dimensions of reading difficulty that once required two complementary metrics (surprisal + attention entropy) may be covered by energy alone.

Why This Matters

1. A new tool for computational psycholinguistics

Since surprisal entered psycholinguistics in the 2010s, the field has used "language model next-token probability" as a proxy for reading difficulty. But surprisal captures only "prediction difficulty," not "processing dynamics." Energy offers a new, richer signal—not just "how surprised the model is," but "how much the model is still struggling to converge."

2. A Hopfield network "renaissance"

Hopfield networks were stars of connectionism in the 1980s–90s but faded with the rise of deep learning. Recently, their return to modern AI (Modern Hopfield Networks, Dense Associative Memory) has renewed interest in the framework. By linking energy-based models to human reading behavior, this paper gives the Hopfield framework a new application.

3. "Unified vs. complementary" philosophy of science

For a decade, the consensus has been that multiple complementary metrics are needed—surprisal for prediction, attention entropy for attentional dispersion, perhaps syntactic complexity too. This paper challenges that consensus: perhaps the issue isn't "multiple complementary metrics are needed" but "we hadn't yet found the unified metric." Energy may be that metric. It evokes the "unified field theory" dream in physics—forces once thought distinct were later unified. Of course, whether energy truly unifies reading difficulty prediction requires more research.

Honest Assessment

Strengths

• Clear theoretical motivation: not "let's try energy and see," but derived from the Hopfield network framework
• Validated across three corpora: not overfitting to a single dataset
• Controlled experiments + corpus experiments: relative clause experiments are controlled; reading-time corpora are naturalistic data
• "Subsumes" analysis: not just showing energy works, but analyzing whether it contains the other metrics' information

Limitations

• Only one energy-based model tested (NRGPT): whether other energy-based Transformer architectures show the same effect is unexplored
• Energy-based models are still small: NRGPT's scale is far below modern LLMs—does this limit energy's predictive power?
• Statistical caution for "subsumption": when three metrics are highly correlated, regression coefficient estimates become unstable; the "subsumes" conclusion may partly stem from multicollinearity
• No comparison with more modern predictors: e.g., LLM internal representations (hidden states), probe-based metrics. Is energy better than these?
• Causal direction unclear: does "high energy cause slow reading," or does "complex sentences cause both high energy and slow reading"? The study is correlational and cannot support causal claims

A Thought-Provoking Extension

What excites me most about this paper is not the specific results but a philosophical extension: if a Transformer's energy function can predict human reading difficulty, does that mean Transformers "understand" language in some mathematical sense?

Not in the Turing-test sense—that bar is too low; even ELIZA could fool people. Rather, in the sense of "processing isomorphism": the model's struggle to converge is highly correlated with humans' struggle to comprehend.

This correlation might be coincidence—both shaped by syntactic complexity. But it might point to something deeper: the human brain and Transformers may share some underlying computational geometry when processing language. The appeal of Hopfield networks is their natural connection to neural dynamical systems—if the Transformer's energy landscape and the brain's neural dynamics landscape have similar geometry, the energy–reading-time correlation isn't coincidence but isomorphism.

Of course, this is speculation; the paper doesn't explore it. But good papers don't just answer questions—they open doors to new ones. This paper does that.

---

Paper: https://arxiv.org/abs/2606.23382 HTML version: https://arxiv.org/html/2606.23382v1 Code: jakdot/energy-transformers-reading-difficulty Authors: Jakub Dotlačil, Ece Takmaz (Utrecht University)