English static mirror for SEO/GEO · AI-assisted translation · Read Chinese original
智柴网 中文原文

Visual Retrieval Heads: How VLMs Locate and Extract Visual Evidence

Forum topic · 小凯 · 2026-08-30

Summary

This paper introduces Visual Retrieval Heads (VRHs), a small subset of attention heads (about 1.7-2.6% of all heads) in vision-language models (VLMs) that are causally responsible for grounding text descriptions to image regions. Inspired by retrieval heads in large language models, the authors unify existing head-scoring methods under a design space spanning query tokens, key aggregation, and cross-sample aggregation, showing that scoring attention from output prediction tokens summed over the ground-truth referent most reliably identifies causal heads. Across 11 VLMs and 5 referring expression benchmarks, masking only the top 20 VRHs reduces localization accuracy by up to 80 percentage points, while masking the same number of random heads has little effect. VRHs generalize across visual referring tasks, remain causal on attribute, spatial, counting, and visual math benchmarks, are functionally specific (preserving output format while breaking localization), and causally transfer between VLMs sharing the same LLM backbone despite differing vision encoders, projectors, and instruction tuning. Paper: arXiv 2608.27417.

Paper Overview

Field: Computer Vision Authors: Chanho Park, Daehyeon Choi, Jihyun Lee, Minhyuk Sung Published: 2026-08-27 arXiv: 2608.27417

Summary

Vision-language models (VLMs) can locate an image region referred to by a text prompt and route the corresponding visual evidence to the output, yet the internal mechanism behind this behavior is not understood. Inspired by retrieval heads in large language models, this paper asks whether VLMs contain an analogous mechanism for visual retrieval.

Key Points

  • Introduces Visual Retrieval Heads (VRHs): a small subset of attention heads (about 1.7-2.6%) that are causally responsible for grounding text descriptions to image regions.
  • Recasts existing head-scoring methods under a unified design space over query tokens, key aggregation, and cross-sample aggregation.
  • Shows that scoring attention from output prediction tokens with a sum over the ground-truth referent region most reliably identifies causal heads.
  • Across 11 VLMs and 5 referring expression benchmarks, masking only the top 20 VRHs reduces localization accuracy by up to 80 percentage points, while masking the same number of random heads has little effect.

    • Properties of VRHs

  • Generalization: VRHs generalize across visual referring tasks and remain causal on attribute, spatial, counting, and visual math benchmarks.
  • Functional specificity: masking VRHs preserves output format while breaking localization.
  • Architectural sharing: VRHs causally transfer between VLMs that share the same LLM backbone, even when vision encoders, projectors, and instruction tuning differ.

    • Links

  • arXiv: https://arxiv.org/abs/2608.27417
---

*Auto-collected on 2026-08-30.*

Tags

#vision-language-models#retrieval-heads#attention-heads#interpretability#visual-grounding#computer-vision#arxiv

This page is an English static mirror generated for search and AI citation. It may be a full translation or structured summary of the Chinese original. Canonical interactive discussion lives on the Chinese page: https://zhichai.net/topic/178634239