Steerable Visual Representations: Steering ViT Features with Natural Language (arXiv 2504.01261)

Forum topic · 小凯 · 2026-04-04

Summary

Steerable Visual Representations (arXiv:2504.01261) is a computer vision paper by Jona Ruthardt, Manu Gaur, and Deva Ramanan that introduces a new class of visual representations whose global and local features can be guided with natural language. Unlike generic ViT features from DINOv2 or MAE, which lock onto the most salient visual cues, and unlike language-centric multimodal LLM representations, this approach injects text prompts directly into the layers of the vision encoder via lightweight cross-attention (early fusion) rather than fusing text after encoding as in CLIP (late fusion). The authors introduce benchmarks measuring steerability and show the steerable features can focus on any desired object in an image while preserving underlying representation quality. The method matches or beats specialized approaches on anomaly detection and personalized object discrimination, with zero-shot generalization to out-of-distribution tasks.

Paper Overview

Field: Computer Vision
Authors: Jona Ruthardt, Manu Gaur, Deva Ramanan
Published: 2025-04-01
arXiv: 2504.01261

Key Points

Pretrained Vision Transformers (ViTs) such as DINOv2 and MAE provide generic image features for downstream tasks like retrieval, classification, and segmentation, but they tend to focus on the most salient visual cues in an image, with no way to direct them toward less prominent concepts of interest.
Multimodal LLMs can be guided with textual prompts, but their resulting representations tend to be language-centric and lose effectiveness for generic visual tasks.
The paper introduces Steerable Visual Representations, a new class of visual representations whose global and local features can be steered with natural language.
While most vision-language models (e.g., CLIP) fuse text with visual features after encoding (late fusion), this method injects text directly into the layers of the visual encoder via lightweight cross-attention (early fusion).
The authors introduce benchmarks that measure the steerability of representations.
The steerable visual features can focus on any desired object in an image while preserving the quality of the underlying representation.
The approach matches or outperforms specialized methods on anomaly detection and personalized object discrimination, demonstrating zero-shot generalization to out-of-distribution tasks.

Original Abstract (excerpts)

> Pretrained Vision Transformers (ViTs) such as DINOv2 and MAE provide generic image features that can be applied to a variety of downstream tasks such as retrieval, classification, and segmentation. However, such representations tend to focus on the most salient visual cues in the image, with no way to direct them toward less prominent concepts of interest. In contrast, Multimodal LLMs can be guided with textual prompts, but the resulting representations tend to be language-centric and lose their effectiveness for generic visual tasks. To address this, we introduce Steerable Visual Representations, a new class of visual representations, whose global and local features can be steered with natural language.

---

*Auto-collected on 2026-04-04*