Show a vision-language model (VLM) a picture of a blue strawberry and ask "What color is this strawberry?" — it will honestly answer "blue."
But if you ask "What color are strawberries usually?" — a question that should call on world knowledge rather than visual evidence — it often still answers "blue."
Visual evidence overrides memorized knowledge. This phenomenon had long been observed at the behavioral level, but nobody had opened the model's black box to see who was fighting whom inside, who won, and how.
A team from Harvard University and the University of Tubingen did exactly that. Using activation patching across three different VLM families, they traced this "vision vs. knowledge" tug-of-war layer by layer, head by head, sublayer by sublayer. The result revealed an asymmetric causal structure so clean it reads almost like an aphorism:
Vision is the default pathway; prior knowledge requires active injection.
2.5% of Attention Heads Do the Work of Remembering "Strawberries Are Red"
The researchers designed two kinds of prompts: visual prompts ("What color is this strawberry?") and prior prompts ("What color are strawberries usually?"), while showing the model a color-conflicting image (e.g., a blue strawberry). They then used activation patching at three granularities — residual stream, attention heads, and MLP sublayers — to localize where the decision happens.
Key finding: grounding of prior knowledge relies on a small set of attention heads in the second half of the network (only 2.5%–4.8%). These heads extract answers from stored world knowledge ("strawberries are red") even when the image in front of the model shows blue.
What happens if you ablate these heads?
Under prior prompts, 68%–96% of predictions flip from "knowledge grounding" to "visual grounding" — the model stops saying "red" and switches to "blue."
But under visual prompts, only 0.8%–7.5% of predictions change.
That's the core of the asymmetry: visual grounding works without these heads, but prior grounding collapses without them. In other words, vision is the default mode; knowledge recall is an "advanced feature" that must be actively switched on.
Routing Heads and Writing Heads: A Sparse Causal Circuit
Finer-grained analysis found that these 2.5%–4.8% of heads are not all the same — they fall into two classes:
Routing heads: regulate information flow, deciding which path visual and knowledge information take through the network. Like railway switches — they don't produce content, but they decide where content goes.
Writing heads: directly project the answer token into the residual stream. Like terminal printers — they translate the decision into final output vocabulary.
Routing heads act first; writing heads execute later. Together they accomplish the act of "prior override."
This division of labor holds across all three VLM families and remains consistent across model scales. The paper calls it a "sparse causal circuit" — achieving the most critical function with the fewest components, which is itself a sign of evolutionary pressure.
Why Does a Blue Strawberry Fool the Model?
Back to the original example. Why does the model get misled by visual evidence even when asked "What color are strawberries usually?" — a question that should invoke knowledge?
The paper's answer: because the visual pathway is the default, and the knowledge pathway requires extra activation. When the prompt doesn't strongly signal "use your memory, not your eyes," visual information automatically flows into the decision and overrides knowledge.
This maps subtly onto human cognition. You see a steaming cup of coffee and ask yourself whether coffee is usually drunk hot or cold — your eyes say "hot," your memory says "hot" — usually no conflict. But if someone shows you a cup of coffee billowing with dry-ice fog (looking hot, actually cold), your eyes deceive you, and you must actively invoke knowledge to override the visual intuition.
The VLM's problem is that its "active invocation" mechanism is extremely sparse — only 2.5%–4.8% of heads do this work. Once those heads are disrupted, ablated, or the prompt isn't strong enough, the model degenerates into a purely visual system.
MLP Sublayers: Amplifiers, Not Routers
The paper also examined the role of MLP sublayers. The conclusion: MLPs show the same directional asymmetry as attention heads, but with much weaker magnitude. This suggests MLPs are amplifiers rather than routers — they reinforce decisions already made but don't actively steer them.
This is an important detail. It means that if you want to fix the "vision overrides knowledge" problem in engineering, you should intervene on attention heads, not MLPs. Attention heads are the decision switches; MLPs are just the volume knob.
Cross-Architecture Consistency
The most impressive aspect of this finding is not the result on any single model, but that it holds across three different VLM families. Different training data, different architectural details, different parameter scales — yet the "vision-default, prior-override" causal structure is consistent.
This hints at a deeper regularity: perhaps this is not an accidental quirk of one architecture, but a tendency of multimodal learning itself. Visual information is high-dimensional, dense, and real-time; knowledge information is low-dimensional, sparse, and delayed. When the two conflict, defaulting to the visual pathway may be a kind of "computational convenience" — the visual signal is naturally stronger, and extra machinery is needed to make it yield.
Engineering Implications
This work has a very direct engineering implication: if you're building a VLM system that needs to "sometimes trust the eyes, sometimes trust the memory," you shouldn't assume the model handles this automatically. You need to protect that 2.5%–4.8% of prior-grounding heads at the architecture level, or explicitly activate them at inference time.
A more radical idea: since vision is the default pathway and knowledge requires active injection, could training-time regularization be designed so that the knowledge pathway also becomes "default"? So the model doesn't need extra attention heads to invoke knowledge, and knowledge and vision compete as equals?
The paper doesn't answer this question, but it provides the causal map needed to answer it. You can't fix a mechanism you don't understand — and now, we have the map.
---
Paper: https://arxiv.org/abs/2606.28273
Code repository: https://github.com/nlietzow/vision-default-prior-override