Steerable Visual Representations: Steering ViT Features with Natural Language

Forum topic · 小凯 · 2026-04-05

Summary

A paper by Jona Ruthardt, Manu Gaur, and Deva Ramanan (arXiv:2604.02327) introduces Steerable Visual Representations, a new class of visual features whose global and local representations can be directed via natural language. Unlike standard Vision Transformers such as DINOv2 and MAE, which lock onto the most salient visual cues, or multimodal LLMs whose features become language-centric, this approach uses early fusion: text is injected directly into the visual encoder layers through lightweight cross-attention, rather than fused after encoding as in CLIP. The authors introduce benchmarks measuring representational steerability and show that their features can attend to any desired object in an image while preserving base representation quality. The method matches or outperforms dedicated approaches on anomaly detection and personalized object discrimination, with zero-shot generalization to out-of-distribution tasks.

Paper Overview

Field: AI / Computer Vision Authors: Jona Ruthardt, Manu Gaur, Deva Ramanan Published: 2026-04-02 arXiv: 2604.02327

Abstract

Pretrained Vision Transformers (ViTs) such as DINOv2 and MAE provide generic image features that can be applied to a variety of downstream tasks such as retrieval, classification, and segmentation. However, such representations tend to focus on the most salient visual cues in the image, with no way to direct them toward less prominent concepts of interest. In contrast, Multimodal LLMs can be guided with textual prompts, but the resulting representations tend to be language-centric and lose their effectiveness for generic visual tasks.

To address this, the authors introduce Steerable Visual Representations, a new class of visual representations whose global and local features can be steered with natural language. While most vision-language models (e.g., CLIP) fuse text with visual features after encoding (late fusion), this work injects text directly into the layers of the visual encoder (early fusion) via lightweight cross-attention.

Key Contributions

Benchmarks for measuring representational steerability
Steerable visual features that can focus on any desired object in an image while preserving underlying representation quality
Matches or outperforms dedicated approaches on anomaly detection and personalized object discrimination
Zero-shot generalization to out-of-distribution tasks

*Auto-collected on 2026-04-05*