Imagine chatting with a friend. You ask: "Do you know what Zhang San's new book is about?"
If your friend knows the book, he tells you directly. If not, an honest person says: "I don't know Zhang San, so I don't know his book."
But there is a subtler move: "I hear there's a new book getting attention lately." — He doesn't know Zhang San's book, but he retreats to a broader, safe formulation. He abandons the specific referent "Zhang San's book" and swaps it for the more general "a new book."
This is a key strategy within the cooperative principle described by linguist Paul Grice in 1975: when a speaker is uncertain about a specific referent, they retreat upward along the specificity hierarchy — from "Zhang San's book" to "a new book" to "a book" — trading informativeness for accuracy.
This is called Gricean Retreat.
The question: do large language models do Gricean retreat?
The Hallucination Dilemma of LLMs
When asked about entities absent from training data, the classic LLM failure mode is fabrication. Instead of saying "I don't know," the model generates plausible-sounding but entirely fictional content.
The traditional explanation: the model doesn't know what it doesn't know. It has no awareness of "knowledge boundaries" for unseen entities, so it confidently fabricates.
But is this explanation right? Does the model really "not know"?
A research team from the University of Colorado Boulder answered this with a set of elegant experiments. The answer is surprising: the model knows, but it doesn't use that knowledge.
Experimental Design
The T-REx Benchmark
The team built a test benchmark on the T-REx knowledge graph, which aligns Wikipedia text with knowledge-graph triples such as "Paris — capital — France."
The key design is synthetic entities: researchers replace real entities with fabricated ones (e.g., swapping "Paris" for a fictional city name like "Zarathustra City"), creating entities that could not appear in training data. This lets them precisely control which entities are inside vs. outside the model's knowledge boundary.
Two Core Questions
For each entity, two questions:
Q1: Do the model's activations encode whether the entity is inside or outside its knowledge boundary?
If the model distinguishes "seen" from "unseen" entities, there is an internal knowledge-boundary signal.
Q2: Do the activations predict whether the model is about to generate a specific or a general reference?
If the model "knows" before generating what specificity level it will produce, the generation strategy is driven by an internal signal, not randomness.
Finding 1: The Model Does Know Its Knowledge Boundary
Using linear probes on layer activations:
- Model activations strongly encode whether an entity appeared in training data.
- For models above 22B parameters, AUROC exceeds 90%.
- Even the smallest models show (weaker) predictive signal.
- The best probing layers are usually mid-network.
- Prediction is stronger for argmax decoding (deterministic decoding forces an earlier "decision")
- Weaker but still significant for multinomial decoding
Interestingly, prediction difficulty varies by relation. The "People→Location" relation predicted worst — because the synthetic entities for this relation happened to leak most often into the Pile pretraining corpus, so the model "thought" it had seen them.
Conclusion: the model's internal representations genuinely distinguish "seen" from "unseen" entities.
Finding 2: The Model Also Knows What It Is About to Generate
The second finding is equally striking. Model activations reliably predict, before generation, whether the upcoming reference will be specific or general.
Finding 3: But the Model Does Not Retreat
Both signals exist. The critical question: are they linked?
No. Testing the model's out-of-the-box generation behavior:
Models overwhelmingly generate specific references, regardless of whether the entity is within the knowledge boundary.
Even for synthetic entities the model has never seen, it produces specific, plausible-sounding but fabricated references. Even when correct general alternatives are offered, it still picks specific ones. The specificity bias persists across context lengths — it is not a context-length problem.
The Nature of the Problem
The paper's deepest insight, in one sentence:
"The substrate for Gricean retreat exists, but the strategy to execute it does not."
In plain terms: the model has all the information needed to retreat, but lacks the strategy to use it.
It's like a person who knows they don't know the answer, and knows they are making things up, but can't stop — because they lack the "stop" action.
From a mechanistic interpretability standpoint this means:
1. Knowledge-boundary signals and generation strategy are decoupled. Two independent internal signals exist — "have I seen this entity?" and "what specificity will I generate?" — but they are not connected. 2. Hallucination is not a "not knowing" problem but a "not using what it knows" problem. The model doesn't need to learn its knowledge boundary — it already has it. It needs to learn to retreat when it doesn't know. 3. Generation strategy is trained, not emergent. The preference for specific references comes from training (especially RLHF), which rewards concrete, confident answers.
Gricean Alignment: A New Direction
The paper proposes a forward-looking concept: Gricean alignment.
Traditional hallucination mitigation is post-hoc: generate text, then detect and correct hallucinations with another model.
Gricean alignment shifts levels: use knowledge-boundary signals during generation to steer strategy. If the model internally knows "I haven't seen this entity," use that signal at generation time to guide a retreat to more general references.
This is not a new model but a new alignment objective: traditional alignment targets "generate useful content"; Gricean alignment targets "retreat to safe formulations under uncertainty."
This contrasts interestingly with Epanorthosis (LLMs systematically reproducing classical rhetorical self-correction). Epanorthosis is "speak first, revise after" — inference-time intervention. Gricean retreat is "know first, retreat before speaking" — a generation-time strategy.
Limitations and an Honest Assessment
The paper candidly discusses limitations:
1. Synthetic entities may not fully represent real scenarios. Real "unknown entities" carry more contextual cues (e.g., semantic hints in names); model behavior may differ. 2. Only linear probes were used. Nonlinear signals may be stronger, though linear probes suffice to make the point. 3. Gricean alignment was not implemented. The paper diagnoses the problem; the alignment concept is a framework, not a method. 4. Limited model scope. Mostly open-source models; closed-source behavior is unknown.
Conceptually, this paper is another instance of the "evaluation blind-spot" pattern: traditional hallucination evaluation asks "is the output correct?" — an output-level check. This paper shows the problem lies deeper: correct internal signals exist, but the generation strategy ignores them. Evaluate outputs only, and you never see this gap.
The "models already know" series gains another entry: models know their knowledge boundary (linear-probe AUROC > 90%) and know their generation strategy (specificity is predictable), yet never use this knowledge to guide generation. As with SOPHIA's residual-stream directions and sycophancy findings, cross-paper consensus is strengthening: models internally know more than their outputs reveal; the challenge is activating and using it.
---
Paper link: https://arxiv.org/abs/2608.13484
HTML version: https://arxiv.org/html/2608.13484v1