Show a vision-language model a picture of a blue strawberry and ask "What color is this strawberry?" It will dutifully answer "blue."
But if you ask "What color are strawberries usually?"—a question that should call on world knowledge rather than visual evidence—it often keeps answering "blue."
Visual evidence overrides remembered knowledge. This phenomenon has long been observed at the behavioral level, but no one had opened the model's black box to see who was fighting whom inside, who won, and how.
A team from Harvard University and the University of Tübingen did exactly that. Using activation patching across three VLM families, they traced this "vision vs. knowledge" tug-of-war layer by layer, head by head, and sublayer by sublayer. The result reveals an asymmetric causal structure, so clean it reads almost like an aphorism:
Vision is the default pathway; prior knowledge requires active injection.
2.5% of Attention Heads Do the Remembering That "Strawberries Are Red"
The team designed two kinds of prompts—visual prompts ("What color is this strawberry?") and prior prompts ("What color are strawberries usually?")—while showing the model a color-conflicting image (e.g., a blue strawberry). They then used activation patching at three granularities—residual stream, attention heads, and MLP sublayers—to localize where the decision happens.
Key finding: grounding in prior knowledge depends on a small set of attention heads in the second half of the network (only 2.5%–4.8%). These heads extract the answer from stored world knowledge ("strawberries are red") even when the image shows blue.
What happens if you ablate these heads?
Under prior prompts, 68%–96% of predictions flip from "knowledge-grounded" to "vision-grounded"—the model stops saying "red" and switches to "blue."
But under visual prompts, only 0.8%–7.5% of predictions change.
That is the core of the asymmetry: vision grounding works without these heads, but prior grounding collapses without them. In other words, vision is the default mode; knowledge recall is an "advanced feature" that must be actively switched on.
Routing Heads and Writing Heads: A Sparse Causal Circuit
Finer analysis shows these 2.5%–4.8% of heads are not all the same. They split into two classes:
Routing heads: regulate information flow, deciding which path visual and knowledge information take through the network. Like railroad switches—they don't produce content, but they decide where content goes.
Writing heads: directly project the answer token into the residual stream. Like terminal printers—they translate the decision into final output vocabulary.
Routing heads act first; writing heads execute later; together they accomplish "prior override."
This division of labor holds across three different VLM families and remains consistent across model scales. The paper calls it a "sparse causal circuit"—accomplishing the most critical function with the fewest components, itself a hallmark of evolutionary pressure.
Why Does a Blue Strawberry Fool the Model?
Back to the original example. Why does a model get led astray by visual evidence on a question like "What color are strawberries usually?"
The paper's answer: because the visual pathway is the default, while the knowledge pathway needs to be switched on. When the prompt doesn't strongly signal "use memory, not your eyes," visual information automatically flows into the decision and overrides knowledge.
This has a subtle parallel in human cognition. Seeing a steaming cup of coffee and asking yourself whether coffee is usually drunk hot or cold, your eyes say "hot" and your memory says "hot"—usually no conflict. But if someone shows you a cup billowing with dry-ice fog (visually like hot, actually cold), your eyes deceive you, and you must actively invoke knowledge to override the visual intuition.
The VLM's trouble is that its "active invocation" mechanism is extremely sparse—only 2.5%–4.8% of heads do this. Once these heads are disrupted, ablated, or the prompt isn't strong enough, the model degenerates into a purely visual system.
MLP Sublayers: Amplifiers, Not Routers
The paper also examined the role of MLP sublayers. The conclusion: MLPs show the same directional asymmetry as attention heads, but with much weaker magnitude. This suggests MLPs are amplifiers rather than routers—they strengthen decisions already made but don't actively steer decision direction.
This is an important detail. It means if you want to fix the "vision overriding knowledge" problem in engineering, you should intervene at the attention heads, not the MLPs. Attention heads are the decision switches; MLPs are just the volume knob.
Cross-Architecture Consistency
The most impressive aspect of this work is not the result on any single model, but that it holds across three different VLM families. Different training data, different architectural details, different parameter scales—yet the "vision default, prior override" causal structure is consistent.
This hints at a deeper regularity: perhaps this is not an accidental property of one architecture, but a tendency of multimodal learning itself. Visual information is high-dimensional, dense, and real-time; knowledge information is lower-dimensional, sparse, and delayed. When they conflict, defaulting to the visual pathway may be a "computational convenience"—visual signals are naturally stronger, and extra machinery is needed to make them yield.
Engineering Implications
This work has a very direct engineering implication: if you're building a VLM system that must "sometimes trust the eyes, sometimes trust memory," you should not assume the model handles this automatically. You need to protect those 2.5%–4.8% prior-grounding heads at the architecture level, or explicitly activate them at inference time.
A more radical idea: since vision is the default pathway and knowledge requires active injection, could training include a regularization that makes the knowledge pathway "default" too—letting knowledge and vision compete as equals rather than requiring special attention heads to recall knowledge?
The paper doesn't answer this question, but it provides the causal map needed to answer it. You cannot fix a mechanism you don't understand—and now we have that map.
---
Paper: https://arxiv.org/abs/2606.28273
Code: https://github.com/nlietzow/vision-default-prior-override