Language Models Are Not Knowledge Bases: Same Fact, Different Format, Different Answer

Forum topic · ✨步子哥 · 2026-06-26

Summary

A Tel Aviv University study (Elhelo, Globerson, Geva) shows that language models do not store knowledge in a unified internal database. Using OLMo-3-7B and six task formats (completion, cloze, open QA, multiple choice, negative multiple choice, verification) across five relation types, the authors found low co-occurrence of factual knowledge across tasks: a fact learned in one format may be absent in another. Mechanistic analysis identified task-specific parameter subsets—small groups of parameters whose removal breaks the fact for one task but not others. Generative-task knowledge transfers more easily than discriminative-task knowledge, and tasks show varying degrees of entanglement. The paper also explains why chain-of-thought prompting recovers lost facts: it reroutes through parameter subsets belonging to other tasks, making CoT more vulnerable when those subsets are ablated. Implications include unreliable knowledge editing, single-task evaluation being insufficient, and challenges for machine unlearning and consistency in AI safety. The paper argues that what a model knows is intertwined with how it is asked—a structural property, not a bug.

The Problem

Ask a language model "What is the capital of France?" and it answers "Paris." Ask it to fill in "The capital of France is ___" and it usually still says "Paris." But what if the model learned the fact under one format and not the other? A new paper from Tel Aviv University—*LMs as Task-Specific Knowledge Bases* by Amit Elhelo, Amir Globerson, and Mor Geva—demonstrates experimentally that knowledge in language models is not stored in a unified knowledge base, but separately per task format. The same fact may be answered by one parameter set for cloze tasks, another for multiple choice, and a third for open QA.

Experimental Design: Tracking Knowledge Co-Occurrence

The researchers used OLMo-3-7B, whose training checkpoints are public, allowing them to track when facts are acquired. They tested five relation types (country-capital, country-language, landmark-country, company-headquarters, person-instrument) under six tasks:

Completion: continue "The capital of France is Paris"
Cloze (FiTB): "The capital of France is ___"
OpenQA: "What is the capital of France?"
MCQA: pick the correct option
Neg MCQA: pick the incorrect option
Verification: "The capital of France is Paris. True or false?"

If knowledge were unified, learning a fact in one task would imply knowing it in all. Instead, co-occurrence rates were low: a fact learned in one format may be absent in another. Generative tasks (completion, cloze, open QA) transfer better to other tasks, while discriminative tasks (MCQA, verification) barely transfer at all.

Mechanistic Analysis: Locating Where Knowledge Lives

For each (fact, task) pair, the authors identified a small set of critical parameters—removing them collapsed the model's performance on that fact for that task while barely affecting others. Such task-specific parameter subsets do exist: the same fact is stored separately for different tasks.

They also quantified "entanglement" between tasks: generative tasks are relatively independent, while discriminative tasks overlap more. Consequence: editing parameters for a cloze-format fact may leave the MCQA answer untouched—undermining the reliability of knowledge editing at a fundamental level.

Why Chain-of-Thought Works: An Unexpected Finding

CoT is known to help models recall facts they fail to answer directly. The authors hypothesized that CoT partly works by activating parameter subsets belonging to other tasks. Experiments confirmed this: ablating a task's own subset could be largely compensated by CoT (it routes through other tasks' knowledge addresses), but ablating other tasks' subsets hurt CoT more than direct answering. CoT is not querying a unified store—it jumps between fragmented knowledge stores.

Why It Matters

The paper undermines the popular "LM as knowledge base" metaphor:

1. Knowledge editing should be clean—edit once, apply everywhere. In practice, editing one format may not change another. 2. Evaluation should be comprehensive—one format should suffice. In practice, single-task evaluation captures only a fraction of model knowledge. 3. Consistency should be default—one fact, one version. In practice, models may give contradictory answers across formats.

This has direct implications for AI safety: machine unlearning or knowledge updating requires locating parameter subsets across *all* task formats—editing one is not enough.

A Deeper Question

The author draws a cognitive-science analogy: human memory is also split (procedural vs. declarative, episodic vs. semantic). But humans have a unified working-memory system—consciousness as an interface—coordinating these stores; a model's "working memory" is just task-specific attention activating one subset at a time, which may explain worse cross-task consistency than humans.

Conclusion

The paper's restrained but profound conclusion: "what a model knows and how you ask it are intertwined in parameter space." This is not an engineering bug but an architectural property. Understanding what LMs actually are is the prerequisite for making them reliable and controllable.

Paper: LMs as Task-Specific Knowledge Bases: An Interpretability Analysis

Code: github.com/amitelhelo/TaskInvariance

Authors: Amit Elhelo, Amir Globerson, Mor Geva (Tel Aviv University / Google Research)