Towards Robustness Against Typographic Attacks with Training-free Concept Localization

Forum topic · 小凯 · 2026-07-06

Summary

A paper by Bohan Liu, Wenqian Ye, Guangzhi Xiong, Zhenghao He, Sanchit Sinha, and Aidong Zhang (arXiv:2607.02494, cs.CV/cs.CL) addresses the typographic attack (TA) vulnerability in CLIP-based vision encoders. CLIP models, which underpin most modern large vision-language models (LVLMs), can be misled by irrelevant text appearing in images, biasing visual representations toward lexical meaning rather than true visual semantics — a serious risk for safety-critical applications such as autonomous driving. The authors propose a novel, training-free mechanistic interpretability method that provides sampling-based interpretations of hidden state representations and quantitatively attributes semantic versus lexical attention to individual heads. Through probability analysis and circuit mining, they isolate specific Vision Transformer components that disproportionately encode lexical information, identifying the mechanical root of typographic attacks. Simple interventions on these identified circuits — such as selectively adjusting attention weights — require no additional training yet significantly improve robustness on targeted classification, outperforming supervised and unsupervised defenses. Applied to the vision encoders of several state-of-the-art LVLMs, the interventions yield notable accuracy gains on RIO-Bench under typographic attack perturbations, confirming the effectiveness and generalization of this mechanistic approach.

Paper Overview

Field: NLP / Vision-Language Authors: Bohan Liu, Wenqian Ye, Guangzhi Xiong, Zhenghao He, Sanchit Sinha, Aidong Zhang Published: 2026-07-02 arXiv: 2607.02494 Categories: cs.CV, cs.CL

Summary

Models trained via Contrastive Language-Image Pretraining (CLIP) serve as the foundational vision encoders for most modern Large Vision Language Models (LVLMs). Despite their widespread adoption, CLIP models exhibit a critical yet underexplored failure mode: irrelevant text appearing within images confounds visual representations, biasing them toward lexical meaning rather than true visual semantics. This robustness issue, commonly described as a Typographic Attack (TA), exposes a vulnerability that poses a significant risk to safety-critical applications such as autonomous driving.

To achieve interpretable and effective robustness against TA, the authors propose a novel, training-free mechanistic interpretability method. The approach:

Provides sampling-based interpretations of hidden state representations
Quantitatively attributes semantic vs. lexical attention to individual attention heads
Uses probability analysis and circuit mining to isolate specific Vision Transformer (ViT) components that disproportionately encode lexical information, identifying the mechanical root of TA

Key Findings

Simple interventions applied directly to the identified circuits — such as selectively adjusting attention weights — significantly improve robustness against typographic attacks in targeted classification, without any additional training.
These interventions outperform both supervised and unsupervised defense methods.
Applying the interventions to the vision encoders of several state-of-the-art LVLMs yields significant accuracy gains on RIO-Bench under typographic attack perturbations in visual question answering.

The results confirm the effectiveness and generalization capability of this mechanistic approach.

---

*Auto-collected on 2026-07-06*

Towards Robustness Against Typographic Attacks with Training-free Concept Localization

Summary

Paper Overview

Summary

Key Findings

Tags