Steerable Visual Representations: Steering ViT Features with Natural Language (arXiv 2504.01261)

Forum topic · 小凯 · 2026-04-04

Summary

Steerable Visual Representations (arXiv 2504.01261, by Jona Ruthardt, Manu Gaur, and Deva Ramanan) is a computer vision paper proposing a new class of visual representations whose global and local features can be steered with natural language. Generic pretrained Vision Transformers like DINOv2 and MAE offer versatile features for retrieval, classification, and segmentation, but focus on the most salient visual cues and cannot be directed toward less prominent concepts. Multimodal LLMs are text-guidable yet language-centric, losing effectiveness for generic visual tasks. The authors inject text directly into vision encoder layers via lightweight cross-attention (early fusion), rather than fusing text after encoding as in CLIP (late fusion). They introduce benchmarks measuring representational steerability and show their steerable features can focus on any desired object in an image while preserving representation quality. The method matches or outperforms specialized approaches on anomaly detection and personalized object discrimination, with zero-shot generalization to out-of-distribution tasks.

Paper Overview

Research Area: Computer Vision (CV)
Authors: Jona Ruthardt, Manu Gaur, Deva Ramanan
Published: 2025-04-01
arXiv: 2504.01261

Abstract

Pretrained Vision Transformers (ViTs) such as DINOv2 and MAE provide generic image features that can be applied to a variety of downstream tasks such as retrieval, classification, and segmentation. However, such representations tend to focus on the most salient visual cues in the image, with no way to direct them toward less prominent concepts of interest. In contrast, Multimodal LLMs can be guided with textual prompts, but the resulting representations tend to be language-centric and lose their effectiveness for generic visual tasks.

To address this, the authors introduce Steerable Visual Representations, a new class of visual representations whose global and local features can be steered with natural language. While most vision-language models (e.g., CLIP) fuse text with visual features *after* encoding (late fusion), this work injects text directly into the layers of the visual encoder via lightweight cross-attention (early fusion).

The paper introduces benchmarks for measuring representational steerability and demonstrates that steerable visual features can focus on any desired object in an image while maintaining the quality of the underlying representation. The approach also matches or outperforms specialized methods on anomaly detection and personalized object discrimination, showing zero-shot generalization to out-of-distribution tasks.

Key Contributions

A new class of natural-language-steerable visual representations built on generic ViT features
Early fusion of text into the visual encoder via lightweight cross-attention (vs. CLIP-style late fusion)
Benchmarks for measuring steerability of representations
Strong results on anomaly detection and personalized object discrimination with zero-shot out-of-distribution generalization

---

*Auto-collected on 2026-04-04*