Have you ever met a student who memorizes every textbook fact perfectly but freezes on exams? They remember each fact, yet cannot use those facts to derive new conclusions.
Large language models have the same problem.
A paper from the University of Stuttgart, *Knowledge That Is Linear Before Fine-Tuning Generalizes After It*, introduces an elegant conceptual distinction: active facts vs. inert facts. Active facts can not only be recalled by the model but also support learning new generalizations after fine-tuning. Inert facts can be recalled accurately too, but fine-tuning simply can't put them to work.
This distinction matters because it reveals an overlooked truth: a model's ability to memorize a fact and its ability to use that fact to learn new knowledge are two entirely different things.
A Clever Experimental Design
The researchers exploited relations already present from pretraining, such as "Paris → Europe" (city-to-continent mapping), then randomly assigned each continent a new label: Europe → crab, Asia → zebra. The new relation becomes "Paris → crab."
They fine-tuned the model on a subset of cities (teaching it "Paris → crab"), then tested generalization to unseen cities: if the model learned "Paris → crab," can it answer "Munich → crab" (since Munich is also in Europe)?
The design's elegance: the new relation is something the model has never seen, but its structure (which cities share a continent) is inherited from the pretraining relation. To generalize, the model can only rely on its internal representation of the original relation. If "city → continent" is well encoded in pretraining, the model generalizes; if poorly encoded, fine-tuning is useless.
Pre-Linearity: The Decisive Variable
The researchers measured a metric called Pre-linearity—the accuracy of a linear probe reading the original relation's answer from the model's representations. In short: is the relation "linearly readable" inside the model?
The results are striking: across 11 models and 70 relations, pre-linearity correlates with post-fine-tuning generalization at Pearson r = 0.87 to 0.92 (median 0.90).
This means: knowing how linearly readable a relation is in the pretrained model lets you predict, with high accuracy, how well the model will generalize on that relation after fine-tuning. The correlation is not an artifact—it holds after controlling for fine-tuning sample size, number of answers, recall accuracy on the original relation, and term frequency in pretraining text.
An even cleverer validation: when the model answered incorrectly after fine-tuning—say, answering "Munich → zebra" with "Munich → Asia" (the original continent name)—these "leaked" original answers appeared more often among the linear probe's top answers than among the model's own top answers. Linear structure predicts what will leak after fine-tuning better than the model's own outputs do.
What This Means
First, not all knowledge is born equal. A model accurately recalling a fact does not mean the fact is "active." Some facts are encoded in the model's linear structure and can be extracted and recombined during fine-tuning; others are recallable but encoded nonlinearly, and fine-tuning simply can't move them.
Second, linear structure is the infrastructure of generalization. Back in 2000, Paccanaro and Hinton proposed that linear relational embeddings let models generalize to unseen relational facts. This paper validates that idea in real LLMs: linear readability isn't just theoretical elegance—it is the practical predictor of generalization.
Third, direct implications for training. If you want a model to generalize well after fine-tuning in some domain, merely exposing it to those facts during pretraining is not enough. You need the facts encoded in linearly readable structures. This could influence pretraining data design and training objectives.
An Analogy
Imagine knowledge as a library. Inert facts are like books dumped randomly in a corner—the librarian can find them (recall is accurate), but with no logical arrangement, you cannot infer where a new book should go from existing ones. Active facts are books neatly organized by the Dewey Decimal System—findable, and their positional relationships let you infer where a missing book belongs.
Linear structure is that classification system. Without it, more books are just a pile of facts; with it, facts gain structural reasoning power.
Honest Limitations
The experiments use synthetic "relational re-labeling" tasks—clever, but distant from real-world fine-tuning, where new knowledge usually involves entirely new reasoning paths rather than simple re-labeling. Whether pre-linearity's strong predictive power transfers to real fine-tuning needs further verification.
Also, correlation is not causation. Pre-linearity strongly predicts generalization, but a third factor (e.g., some structural feature of training data) could cause both. The researchers controlled for multiple confounders, but fully establishing the causal chain requires more work.
Closing Thoughts
What I admire most about this paper is that it proposes a correct distinction. In AI research, asking the right question often matters more than giving the answer. "What does the model know" and "what can the model do with what it knows" are different questions, and the active-vs-inert facts distinction refines the latter: not all recallable knowledge supports generalization—linear readability is the key structural factor.
This echoes a recent wave of research—from "composition failure" (can read + can compute ≠ can compute while reading) to "soundness ≠ correctness" (type checking catches execution errors but not task errors). The AI community is increasingly dissecting the black box of "model capability," and each dissection reveals that surface-level "capability" is composed of multiple independent dimensions.
Paper: https://arxiv.org/abs/2610.22300
Code: https://github.com/NightMachinery/active-inert-facts