Language Models Aren't Knowledge Bases: Same Fact, Different Prompt Format, Different Answer

An Awkward Question

Ask a language model: "What is the capital of France?" It answers "Paris." Change the format: "The capital of France is ___." It still answers "Paris."

Looks fine. But what if the model only learned the first format during training and not the second? The same fact, phrased differently, can produce different answers—or none at all.

This isn't hypothetical. In the paper *LMs as Task-Specific Knowledge Bases*, Amit Elhelo, Amir Globerson, and Mor Geva of Tel Aviv University demonstrate experimentally that knowledge in language models is not stored in a unified "knowledge base" but is stored separately per task format. The same fact is answered by one set of parameters in fill-in-the-blank, another in multiple choice, and yet another in open-ended QA.

It's as if your brain stored "Paris is the capital of France" once for conversation, once for exam blanks, and once for true/false questions—with no communication between the three copies.

Experimental Design: Tracking Knowledge "Co-occurrence"

The researchers used OLMo-3-7B, whose public checkpoints allow tracking of capability changes across training. They tested five relation types (country–capital, country–language, landmark–country, company–HQ city, person–instrument) across six task formats:

• Completion: continuing "The capital of France is Paris"
• Fill-in-the-blank (FiTB): "The capital of France is ___"
• OpenQA: "What is the capital of France?"
• MCQA: choose the correct option among four
• Negated MCQA: choose the incorrect option
• Verification: "The capital of France is Paris. True or false?"

If the model had a unified knowledge base, learning a fact would transfer to all six tasks simultaneously. Instead, co-occurrence was low: a fact learned in one task might be completely absent in another.

Crucially, knowledge from generative tasks (completion, FiTB, OpenQA) transfers more easily to other tasks, while knowledge from discriminative tasks (MCQA, verification) barely transfers at all. You may fluently say Paris is the capital of France, yet pick London on a multiple-choice test—because your "selection" knowledge and "expression" knowledge live in different places.

Mechanistic Analysis: Finding Knowledge's "Address"

Behavioral results were only the first step. The authors ran mechanistic analyses: for each (fact, task) pair, they located a small set of critical parameters whose removal collapses performance on that fact in that task, with little effect on other facts or tasks.

Such parameter subsets do exist. The same fact corresponds to different parameter subsets across tasks. The model doesn't store "Paris is the capital of France" once and reuse it; it stores a copy per phrasing.

The authors also quantified "entanglement" across tasks: generative tasks show low entanglement (independent stores), discriminative tasks high entanglement (overlapping stores). This means editing a fact via one task's parameters may leave answers in other tasks untouched—fundamentally undermining the reliability of knowledge editing.

Why Does Chain-of-Thought Work? An Unexpected Finding

It's known that chain-of-thought (CoT) prompting helps models recall facts they can't retrieve directly. Why? The researchers hypothesized that CoT works partly by activating the parameter subsets of *other* tasks.

Experiments confirmed this: deleting a task's own parameter subset, CoT recovers most of the performance loss by detouring through other tasks' "knowledge addresses." But deleting other tasks' subsets hurts CoT more than direct answering—CoT depends heavily on those bypass stores.

Like reconstructing an answer through reasoning when you've forgotten the standard one, CoT isn't querying a unified knowledge base—it's hopping among fragmented knowledge stores.

Why This Matters

The paper undermines the popular metaphor "language models are knowledge bases." If they were:

1. Knowledge editing would be clean—change one place, everywhere updates. In reality, editing fill-in-the-blank knowledge may not change MCQA answers. 2. Knowledge evaluation would be comprehensive—one phrasing would suffice. In reality, single-task evaluation sees only a corner of the model's knowledge. 3. Knowledge consistency would be the default—the same fact shouldn't have conflicting versions. In reality, models can give contradictory answers across tasks.

This has direct implications for AI safety and control. Machine unlearning of a sensitive fact requires locating all task-specific parameter subsets—changing one isn't enough. Knowledge updates (e.g., changing "US President") must cover all task formats.

A Deeper Question

The paper invites a cognitive science analogy: human memory is also non-unified—procedural vs. declarative, episodic vs. semantic. The model's "task-specific encoding" may be a similar differentiation, with specialized parameter pathways per computational operation.

But humans have a unified working memory coordinating these stores; a model's "working memory" is just the attention mechanism of the current task, activating one subset at a time. This may explain why models are worse than humans at cross-task consistency: consciousness serves as a unified interface for people; models have none.

Conclusion

The paper's conclusion is restrained but profound: *what a model knows and how you ask it are intertwined in parameter space*. This isn't an engineering bug—it's an architectural property. Language models are not knowledge bases, and perhaps should never aspire to be—but understanding what they actually are is the prerequisite for making AI reliable and controllable.

Next time you ask a language model a question, remember: your phrasing determines which part of its "memory" answers.

---

Paper: LMs as Task-Specific Knowledge Bases: An Interpretability Analysis

Code: github.com/amitelhelo/TaskInvariance

Authors: Amit Elhelo, Amir Globerson, Mor Geva (Tel Aviv University / Google Research)