Gaze Heads: How Vision-Language Models Look at What They Describe — Paper Explainer

Forum topic · 小凯 · 2026-06-15

Summary

This post explains the paper 'Gaze Heads: How VLMs Look at What They Describe' by Gandikota and Bau. Using comic-strip images with clear spatial narrative order, the authors identify a small subset of attention heads (about 5-9% of all heads) in vision-language models such as LLaVA-1.5, Qwen2-VL, and InternVL2 whose attention reliably tracks the image region the model is currently describing. By forcibly masking these 'gaze heads' toward a chosen image region at inference time, researchers steer the model's descriptions with up to 83.1% success, while random heads have near-zero effect. The mechanism appears across model families and scales (2B-32B parameters), concentrates in middle layers, and transfers from comics to natural COCO images. Models trained with frozen visual encoders lacked clear gaze heads, suggesting end-to-end training enables this emergent gaze behavior. The finding offers a training-free steering tool, a diagnostic method for debugging visual grounding, and applications in medical imaging, video description, and accessibility.

Gaze Heads: How VLMs Look at When They Describe — Paper Explainer

A Chinese forum post explains *Gaze Heads: How VLMs Look at What They Describe* by Rohit Gandikota and David Bau, a mechanistic interpretability study of vision-language models (VLMs).

Background

VLMs like LLaVA, Qwen-VL, and InternVL encode images as patch tokens, insert them into a language model's token sequence, and generate text. The paper asks: when the model describes "a cat on the sofa," which parts of the image do its internal attention heads actually attend to?

Key points

Discovery of gaze heads: A small subset of attention heads — roughly 5–9% of all heads (e.g., 32–58 of 1,280 heads in LLaVA-1.5-7B) — show attention patterns that track the image region currently being described.
Experimental design: Comic strips provide spatially ordered narratives (read left-to-right, top-to-bottom), letting researchers define a ground-truth "gaze target" per panel. Heads were scored by the correlation between attention mass and the target region.
Causal intervention: By forcing the top-100 gaze heads' attention onto a chosen image region during generation, the model's descriptions follow that region with 83.1% success. Randomly selected heads have near-zero steering effect; intervening on all heads makes output collapse.
Generalization: The same intervention works on natural COCO images without grid structure, steering the model toward specific objects or regions. Dynamic mid-sentence gaze shifts produce smooth, token-level transitions in descriptions.
Universality across models and scales:

| Model | Params | Gaze-head share | Intervention success | |-------|--------|-----------------|----------------------| | LLaVA-1.5 | 7B | ~7% | 83.1% | | LLaVA-1.5 | 13B | ~6% | ~80% | | Qwen2-VL | 2B | ~5% | ~75% | | Qwen2-VL | 7B | ~6% | ~82% | | InternVL2 | 8B–32B | ~6–8% | ~78–81% |

Layer distribution: Gaze heads concentrate in middle layers (roughly layers 10–25 of a 32-layer model), rarely appear in the earliest layers, and taper off in the deepest layers — consistent with a cross-modal alignment role.
Functional specialization: Some heads respond more to salient objects, others to spatial position, a few to text regions; overall they act as dynamic spatial pointers rather than content detectors.
Exception: Some model families using frozen visual encoders (training only the connector) lack identifiable gaze heads, suggesting the mechanism emerges from end-to-end joint training.

Why it matters

Training-free steering: Unlike fine-tuning, prompt engineering, or RLHF, gaze-head interventions directly control model behavior at inference time with no retraining.
Debugging and auditing: Comparing where gaze heads point versus the model's answer helps localize whether errors stem from visual understanding or language generation.
Potential applications: Precision control in medical imaging VQA, temporally aligned video description, and interactive image description for accessibility (e.g., "tell me what's in the top-left corner").

Reference

Gandikota, R., & Bau, D. Gaze Heads: How VLMs Look at What They Describe. arXiv preprint arXiv:2606.14703.