Steerable Visual Representations: Steering Vision Transformer Features with Natural Language

Forum topic · 小凯 · 2026-04-05

Summary

A new paper (arXiv:2604.02327) by Jona Ruthardt, Manu Gaur, and Deva Ramanan introduces Steerable Visual Representations, a novel class of visual features that can be guided with natural language. Unlike pretrained Vision Transformers such as DINOv2 and MAE, whose features focus on salient visual cues and cannot be directed toward less prominent concepts, and unlike language-centric multimodal LLM representations, this approach injects text directly into the layers of the visual encoder via lightweight cross-attention (early fusion), rather than fusing text and vision after encoding as in CLIP-style late fusion. The authors introduce benchmarks measuring representational steerability, showing the steerable features can focus on arbitrary desired objects in an image while preserving base representation quality. The method matches or outperforms dedicated approaches on anomaly detection and personalized object discrimination, with zero-shot generalization to out-of-distribution tasks.

Paper Overview

Field: AI / Computer Vision Authors: Jona Ruthardt, Manu Gaur, Deva Ramanan Published: 2026-04-02 arXiv: 2604.02327

Original Abstract

Pretrained Vision Transformers (ViTs) such as DINOv2 and MAE provide generic image features that can be applied to a variety of downstream tasks such as retrieval, classification, and segmentation. However, such representations tend to focus on the most salient visual cues in the image, with no way to direct them toward less prominent concepts of interest. In contrast, Multimodal LLMs can be guided with textual prompts, but the resulting representations tend to be language-centric and lose their effectiveness for generic visual tasks. To address this, we introduce Steerable Visual Representations, a new class of visual representations, whose global and local features can be steered with natural language. While most vision-language models (e.g., CLIP) fuse text with visual features after encoding (late fusion), we inject text directly into the layers of the visual encoder (early fusion) via lightweight cross-attention. We introduce benchmarks for measuring representational steerability, and demonstrate that our steerable visual features can focus on any desired objects in an image while preserving the underlying representation quality. Our method also matches or outperforms dedicated approaches on anomaly detection and personalized object discrimination, exhibiting zero-shot generalization to out-of-distribution tasks.

Key Points

Problem: Generic ViT features (DINOv2, MAE) are powerful but fixated on salient visual cues; multimodal LLM features are steerable via text prompts but lose effectiveness for generic visual tasks.
Approach: Steerable Visual Representations enable both global and local features to be steered with natural language.
Architecture choice: Instead of late fusion (as in CLIP), text is injected directly into visual encoder layers via lightweight cross-attention (early fusion).
Evaluation: New benchmarks measure representational steerability; steerable features can attend to any desired object while maintaining base representation quality.
Downstream results: Matches or beats dedicated methods on anomaly detection and personalized object discrimination, with zero-shot generalization to out-of-distribution tasks.

---

*Auto-collected on 2026-04-05*