Towards Robustness against Typographic Attack with Training-free Concept Localization

Forum topic · 小凯 · 2026-07-06

Summary

CLIP-based vision encoders underpin most modern large vision-language models (LVLMs), but they are vulnerable to typographic attacks (TA): irrelevant text inside images biases visual representations toward lexical meaning instead of true visual semantics, a risk for safety-critical applications like autonomous driving. This paper (arXiv:2607.02494) by Bohan Liu, Wenqian Ye, Guangzhi Xiong, Zhenghao He, Sanchit Sinha, and Aidong Zhang proposes a novel, training-free mechanistic interpretability method that provides sampling-based interpretations of hidden state representations and quantitatively attributes semantic versus lexical attention to individual attention heads. Using probabilistic analysis and circuit mining, the authors isolate specific Vision Transformer (ViT) components that disproportionately encode lexical information, identifying the mechanistic root of typographic attacks. Simple interventions on the identified circuits—such as selectively adjusting attention weights—require no additional training yet substantially improve robustness in targeted classification, outperforming both supervised and unsupervised defenses. Applying the interventions to vision encoders of several state-of-the-art LVLMs yields significant gains in visual question answering accuracy under TA perturbations on RIO-Bench, confirming the method's effectiveness and generalization.

Paper Overview

Field: NLP / Vision-Language Models Authors: Bohan Liu, Wenqian Ye, Guangzhi Xiong, Zhenghao He, Sanchit Sinha, Aidong Zhang Published: 2026-07-02 arXiv: 2607.02494 Categories: cs.CV, cs.CL

Problem

Models trained via Contrastive Language-Image Pretraining (CLIP) serve as the foundational vision encoders for most modern Large Vision Language Models (LVLMs). Despite their widespread adoption, CLIP models exhibit a critical yet underexplored failure mode: irrelevant text appearing within images confounds visual representations, biasing them toward lexical meaning rather than true visual semantics. This robustness issue, commonly described as a Typographic Attack (TA), poses a significant risk to safety-critical applications such as autonomous driving.

Key Points

The authors propose a novel, training-free mechanistic interpretability method for achieving interpretable and effective robustness against TA.
The method provides sampling-based interpretations of hidden state representations and quantitatively attributes semantic versus lexical attention to individual attention heads.
Through probabilistic analysis and circuit mining, they isolate specific Vision Transformer (ViT) components that disproportionately encode lexical information, identifying the mechanistic root of TA.
Simple interventions applied directly to the identified circuits—such as selectively adjusting attention weights—require no additional training yet significantly improve robustness in targeted classification against typographic attacks.
These interventions outperform both supervised and unsupervised defense methods.
Applying the interventions to vision encoders of several state-of-the-art LVLMs yields significant improvements in visual question answering accuracy under TA perturbations on RIO-Bench.
The results confirm the effectiveness and generalization capability of the mechanistic approach.

Source

arXiv link: https://arxiv.org/abs/2607.02494

*Auto-collected on 2026-07-06.*