Visual Retrieval Heads: How Vision-Language Models Locate and Extract Visual Evidence

Forum topic · 小凯 · 2026-08-30

Summary

A paper by Chanho Park, Daehyeon Choi, Jihyun Lee, and Minhyuk Sung (arXiv:2608.27417) introduces Visual Retrieval Heads (VRHs), a small subset of attention heads (about 1.7-2.6%) in vision-language models that are causally responsible for grounding text descriptions to image regions. Inspired by retrieval heads in large language models, the authors recast existing head-scoring methods under a unified design space spanning query tokens, key aggregation, and cross-sample aggregation, showing that scoring attention from output prediction tokens with a sum over the ground-truth referent region most reliably identifies causal heads. Across 11 VLMs and 5 referring expression benchmarks, masking just the top 20 VRHs degrades localization accuracy by up to 80 percentage points, while masking the same number of random heads has little effect. VRHs generalize across visual referring tasks, remain causal on attribute, spatial, counting, and visual-math benchmarks, are functionally specific (preserving output format while breaking localization), and transfer causally between VLMs sharing the same LLM backbone despite differing vision encoders, projectors, and instruction tuning.

Paper Overview

Field: Computer Vision (CV)
Authors: Chanho Park, Daehyeon Choi, Jihyun Lee, Minhyuk Sung
Published: 2026-08-27
arXiv: 2608.27417

Abstract

Vision-language models (VLMs) can locate an image region referred to by a text prompt and route the corresponding visual evidence to the output, yet the internal mechanism behind this behavior is not understood. Inspired by retrieval heads in large language models, the authors ask whether VLMs contain an analogous mechanism for visual retrieval. They answer affirmatively by introducing Visual Retrieval Heads (VRHs), a small subset of attention heads (about 1.7-2.6%) that are causally responsible for grounding text descriptions to image regions.

To find them, the paper recasts existing head-scoring methods under a unified design space over query tokens, key aggregation, and cross-sample aggregation. The results show that scoring attention from output prediction tokens with a sum over the ground-truth referent region most reliably identifies causal heads.

Key Findings

Across 11 VLMs and 5 referring expression benchmarks, masking only the top 20 VRHs reduces localization accuracy by up to 80 percentage points, while masking the same number of random heads has little effect.
VRHs generalize across visual referring tasks and remain causal on attribute, spatial, counting, and visual-math benchmarks.
VRHs are functionally specific: masking them disrupts localization while preserving output format.
VRHs are architecturally shared: they transfer causally between VLMs that share the same LLM backbone, even when vision encoders, projectors, and instruction tuning differ.