The Geometry of Metaphor: How LLMs Draw a Line Between Literal and Figurative Meaning

Forum topic · 小凯 · 2026-10-24

Summary

Researchers at the University of Amsterdam (Shihui Li, Mette Andersen, Martha Lewis) show that large language models encode metaphor as a simple linear direction in their hidden state space. Using activation patching and Distributed Alignment Search (DAS) across seven models (Gemma, LLaMA, Mistral, Mixtral, Qwen) and three metaphor datasets, the study finds that metaphorical vs. literal usage is linearly separable by the first two principal components, with sentence-final punctuation unexpectedly carrying metaphor information. DAS identifies 16-32 dimensional subspaces that causally control metaphor judgment, achieving intervention interpretability accuracy up to 0.843 on Qwen-72B. Steering interventions flip over 90% of incorrect classifications on Qwen-72B and can bias generation toward metaphorical or literal continuations, with literal-to-metaphor steering proving easier than the reverse. Notably, metaphor encoding quality is not monotonically related to model scale. The findings suggest LLMs represent abstract semantic contrasts as geometric directions rather than surface pattern matching, opening applications in writing assistance, content moderation, and style transfer.

A Secret Inside a Period

Consider two sentences:

"The skin is dry." — a physiological description
"The wit is dry." — a metaphor

Both use the word "dry," but one is literal and one is figurative. Humans distinguish them effortlessly. A language model, however, must encode these two meanings as different internal states somewhere, somehow. What does that look like?

In the paper *The Geometry of Metaphor: Probing and Steering Metaphor Representations in Large Language Models* (arXiv: 2610.31520), Shihui Li, Mette Andersen, and Martha Lewis of the University of Amsterdam offer a surprising answer: metaphor is not a complex higher-order cognitive process — it is a line.

More precisely, it is a direction in hidden state space that is linearly separable by the first two principal components of a PCA.

Finding the "Metaphor Switch"

The team used two classic interpretability tools to probe LLM internals:

Activation Patching: transplanting internal activations from one sentence into another to observe output changes. For example, patching activations from "The skin is dry." onto "The wit is dry." to see whether the model treats the metaphor as literal.

Distributed Alignment Search (DAS): finding a low-dimensional subspace of the hidden states specifically responsible for the concept of "metaphoricity." DAS trains a projection matrix that maps high-dimensional hidden vectors into a small 16- or 32-dimensional space, testing whether that subspace suffices to control the model's metaphor judgment.

Experiments covered seven models — Gemma-9B, LLaMA-8B, LLaMA-70B, Mistral-7B, Mixtral-8×7B, Qwen-7B, Qwen-72B — and three datasets: NOUN (noun metaphors, e.g., "the puzzle is rocket science"), ADJ (adjective metaphors, e.g., "dry wit"), and DOM (domain-specific adjectives).

Activation patching results were clear. Across all models and tasks, three positions had the largest effect on outputs:

Group (a): the differing words in the sentence (e.g., "skin" vs. "wit") — expected, since different words carry different information.
Group (c): the last token (usually ":") — also expected, since its hidden state directly determines next-token prediction.
Group (b): the period "." and the token before it — an unexpected finding.

The surprise is that the period carries no semantic content; it is mere punctuation. Yet the model seems to encode "is this whole sentence a metaphor?" at that position — as if it stamps the final period with a mark: "metaphor" or "literal."

Two Lines in the PCA Plot

If Group (b) really encodes metaphoricity, plotting those hidden states should reveal structure. PCA showed that the first two principal components (PC1 and PC2) linearly separate metaphorical from literal sentences.

Not a fuzzy cluster, not a distribution requiring a complex classifier — a single straight line suffices. On the NOUN dataset, metaphors and literals sit on opposite sides of PC1; on the ADJ dataset, apt and inapt paraphrases are also linearly separable in the PC1–PC2 plane.

This means metaphoricity is a "direction" in the LLM's internal representation, not a "region." The model does not map metaphors to one corner of the space and literals to another; it scores each sentence along a specific axis — positive for metaphor, negative for literal.

This matches earlier findings on truthfulness and sentiment: LLMs tend to encode binary semantic contrasts as linear directions.

DAS: A 16-Dimensional "Metaphor Knob"

While PCA is observation, DAS is intervention. DAS trains a projection matrix into a 16- or 32-dimensional subspace and asks: if I change only the values in this subspace, can I control the model's metaphor judgment?

The answer is yes. DAS intervention interpretability accuracy (IIA) scores:

| Model | Layer | Subspace dim | IIA | |------|-----|-----------|------| | Qwen-72B | 52 | 32 | 0.843 | | Qwen-7B | 14 | 16 | 0.830 | | Gemma-9B | 19 | 16 | 0.807 | | LLaMA-8B | 11 | 16 | 0.793 | | Mistral-7B | 10 | 32 | 0.711 | | Mixtral-8×7B | 11 | 16 | 0.691 | | LLaMA-70B | 28 | 32 | 0.640 |

Higher IIA means the subspace independently controls metaphor judgment. Qwen-72B reaches 0.843 — a 32-dimensional subspace alone steers literal-vs-metaphor processing with 84.3% accuracy.

An interesting finding: larger models do not necessarily have better metaphor subspaces. LLaMA-70B scores only 0.640, lower than its 8B sibling. Qwen-72B is an exception (0.843), but overall there is no monotonic relationship between clarity of metaphor encoding and scale. This suggests metaphor may not be an ability that emerges purely from parameter count, but depends on training data and architecture.

Turning the Knob: From Observation to Control

If metaphor is truly a direction, we should be able to push the model along it — turning literal sentences into metaphors or vice versa. Two control experiments:

Classification flipping: take samples the model originally classified incorrectly, then apply mass-mean or DAS interventions to see if they can be corrected.

Results (DOM dataset, initially incorrect samples):

| Model | Mass-mean flip rate | DAS flip rate | |------|-----------------|-----------| | Qwen-72B | 88.0% | 91.0% | | Gemma-9B | 75.0% | 86.0% | | LLaMA-8B | 54.0% | 71.0% | | Mixtral-8×7B | 72.0% | 61.0% | | Qwen-7B | 62.0% | 61.0% | | LLaMA-70B | 57.0% | 65.0% | | Mistral-7B | 64.0% | 45.0% |

DAS intervention on Qwen-72B flips 91% of incorrect judgments to correct — a genuine "metaphor knob" whose rotation changes how the model understands metaphors.

Generation control: even more striking, can interventions steer generation? Given a literal sentence opening, can we make the model continue metaphorically, and vice versa?

For LLaMA-8B at intervention strength α=5, literal-to-metaphor flip rates were 0.44 (mass-mean) and 0.34 (DAS); metaphor-to-literal were 0.03 and 0.09. The asymmetry is expected: "downgrading" a metaphor to literal only requires suppression, while "upgrading" literal into metaphor requires creating novel figurative expressions.

What This Line Means

The paper's findings compress into one sentence: in an LLM's hidden state space, metaphor is a line.

This line answers a long-standing question: do LLMs "truly understand" metaphor, or merely do surface pattern matching? The answer lies between. LLMs lack the human metaphorical cognition process (no construction from embodied experience to abstract mapping), yet they are not doing surface matching either — they encode metaphoricity as an extractable, intervenable geometric structure that causally determines outputs.

This contrasts interestingly with the "embodied cognition" hypothesis. Lakoff and Johnson, in *Metaphors We Live By*, argued metaphor is a fundamental structure of human thought rooted in bodily experience. LLMs have no bodies, yet spontaneously develop geometric representations of metaphor — perhaps statistical regularities alone suffice to discover systematic differences between literal and extended usage.

The practical implications lie in controllability. If you can find the "metaphor direction," you can:

Writing assistance: automatically turn dry technical documentation into livelier metaphorical language.
Content moderation: detect and adjust inappropriate or biased metaphors in model outputs.
Style transfer: raise or lower the "metaphor density" of a text, like image style transfer.

The authors honestly list limitations: experiments cover only English metaphors; other languages may have entirely different metaphor geometries. Datasets cover only simple sentence-level metaphors — paragraph- and discourse-level metaphors are far more complex. Strongest closed-source models could not be tested.

A Deeper Question

The paper raises a more fundamental question: if metaphor is a line, what isn't a line?

Over the past two years, mechanistic interpretability research has found that truthfulness is a line, sentiment is a line, toxicity is a line — and now metaphor is too. Perhaps LLM internals are more "geometric" than we assumed: not a chaotic black box, but a high-dimensional semantic space composed of linear directions, each corresponding to a human-interpretable concept.

If so, "understanding LLMs" may be more feasible than expected: no need to reverse-engineer the whole network — just find the key directions. Metaphor is only one of them.

---

Paper: The Geometry of Metaphor: Probing and Steering Metaphor Representations in Large Language Models

Authors: Shihui Li, Mette Andersen, Martha Lewis (University of Amsterdam)

Published: 2026-10-22