Gaze Heads: How Vision-Language Models Look at What They Describe - Paper Explained

Forum topic · 小凯 · 2026-06-15

Summary

This in-depth analysis covers a research paper by Rohit Gandikota and David Bau that identifies specialized attention heads, called Gaze Heads, inside vision-language models (VLMs) such as LLaVA, Qwen2-VL, and InternVL2. Using comic strips where narrative order maps to spatial position, the researchers found that roughly 5-9% of attention heads align their attention with the image region currently being described. Crucially, causal intervention experiments show that forcing these heads' attention to a chosen region steers the model's descriptions there, with success rates around 83.1% on comic strips and similarly strong results on natural COCO images, while random heads fail and intervening on all heads collapses output. Gaze Heads appear consistently across model scales from 2B to 32B parameters, cluster in mid-level layers, and emerge primarily from end-to-end training rather than frozen-encoder pipelines. The finding enables training-free, inference-time control of where VLMs look, with applications in medical imaging, video description, model debugging, and accessibility, marking a concrete example of mechanistic interpretability yielding practical control tools.

Gaze Heads: How VLMs Look at What They Describe — Paper Explained

> An analysis of *Gaze Heads: How VLMs Look at What They Describe* by Rohit Gandikota and David Bau (arXiv preprint arXiv:2606.14703).

The Core Question

When a vision-language model (VLM) — such as LLaVA, Qwen-VL, or InternVL — generates a description of an image, where does it "look"? Although these models have no eyes, the authors discovered that a small set of attention heads, termed Gaze Heads, track the image region the model is currently describing. Better yet: control these heads, and you control what the model describes.

Background: Attention as a Gaze

VLMs process images by splitting them into patches (e.g., 14×14 pixels), encoding them with a ViT, and inserting the resulting image tokens into a language model's token stream. Within each Transformer layer, multiple attention heads compute weights deciding which tokens to attend to. The paper asks: when the model writes "a cat on the sofa," do the heads attending at "cat" look at the cat region of the image, or just at previous words?

Finding the Gaze

The Comic Strip Experiment

Natural images are too ambiguous to define where a model "should" look, so the authors used comic strips, where narrative order maps cleanly to spatial order (left-to-right, top-to-bottom). For each attention head, they computed a correlation coefficient between the head's attention distribution and the panel currently being described.

Result: roughly 5–9% of attention heads show strong gaze behavior. In LLaVA-1.5-7B, that means about 32–58 heads out of 1,280 (40 layers × 32 heads).

Causal Intervention

To prove causation, the researchers forced the top-100 gaze heads' attention onto a specific image region during generation. The model then described that region even when it should have described another — with a 83.1% success rate. Control experiments:

Randomly selected 100 heads: near-zero success
Intervening on all heads: model output collapses

Beyond Comics: Generalization

Natural images (COCO): the same intervention successfully steers descriptions to arbitrary regions of everyday photos, e.g., ignoring a central elephant to describe a bird in a corner. Gaze Heads are a general, cross-domain mechanism.
Dynamic gaze shifts: switching the gaze target mid-generation causes the model to finish the current region within a few tokens, then smoothly transition — indicating real-time attention guidance.

Consistency Across Models and Scales

| Model | Params | Gaze head share | Intervention success | |-------|--------|-----------------|----------------------| | LLaVA-1.5 | 7B | ~7% | 83.1% | | LLaVA-1.5 | 13B | ~6% | ~80% | | Qwen2-VL | 2B | ~5% | ~75% | | Qwen2-VL | 7B | ~6% | ~82% | | InternVL2 | 8B | ~8% | ~78% | | InternVL2 | 26B | ~7% | ~81% | | InternVL2 | 32B | ~6% | ~79% |

Key observations:

1. Scale consistency from 2B to 32B parameters (5–9% of heads) 2. Stable intervention success (75–83%) across families 3. Architecture independence: LLaVA (LLaMA-based), Qwen-VL, and InternVL all show the mechanism 4. Caveat: some frozen-encoder model families show no identifiable gaze head set, suggesting gaze emerges from end-to-end joint training of vision and language components.

Anatomy of Gaze Heads

Layer distribution: Gaze Heads concentrate in middle layers (~layers 10–25 of 32), are rare in the first 5–8 layers, and decrease in the top layers — fitting their role in cross-modal alignment rather than feature extraction or text generation.
Functional specialization: some heads favor salient objects, some pure spatial position, a few text regions (signs, labels). They behave more like dynamic pointers than static content detectors.

Why It Matters

Gaze Heads offer a training-free, inference-time control method that complements fine-tuning, prompt engineering, and RLHF. Potential applications:

Precise VQA control, e.g., forcing a medical imaging model to attend to a suspicious region of an X-ray
Frame-aligned video description via per-frame gaze control
Model debugging/auditing: if a gaze head points at the right region but output is wrong, the failure is in language generation, not vision
Interactive accessibility tools, e.g., answering "what's in the top-left corner?" by directly steering gaze

Conclusion

The study shows mechanistic interpretability translating directly into practical control: AI's "gaze" can be located, measured, and steered. As Feynman put it, "What I cannot create, I do not understand" — and now we can not only understand but control how a model sees.

References

Gandikota, R., & Bau, D. (2026). Gaze Heads: How VLMs Look at What They Describe. *arXiv preprint arXiv:2606.14703*.
Liu, H., et al. (2024). Visual Instruction Tuning. *NeurIPS*.
Bai, J., et al. (2023). Qwen-VL: A Frontier Large Vision-Language Model with Versatile Abilities. *arXiv:2308.12966*.
Chen, Z., et al. (2024). How Far Are We to GPT-4V? *arXiv:2404.16821*.
Meng, K., et al. (2022). Locating and Editing Factual Associations in GPT. *NeurIPS*.
Hernandez, E., et al. (2023). Linearity of Relation Decoding in Transformer Language Models. *ICLR*.