Towards Robustness against Typographic Attack with Training-free Concept Localization

Forum topic · 小凯 · 2026-07-06

Summary

This paper addresses the typographic attack (TA) vulnerability in CLIP vision encoders, which underpin most modern large vision-language models (LVLMs). Irrelevant text appearing in images can confound CLIP's visual representations, biasing them toward lexical meaning rather than true visual semantics—a risk for safety-critical applications such as autonomous driving. The authors propose a novel, training-free mechanistic interpretability approach that provides sampling-based interpretations of hidden state representations and quantitatively attributes semantic versus lexical attention to individual attention heads. Using probabilistic analysis and circuit mining, they isolate specific Vision Transformer components that disproportionately encode lexical information, identifying the mechanical root cause of TA. Simple interventions—such as selectively adjusting attention weights—applied directly to the identified circuits significantly improve robustness against typographic attacks in targeted classification, without any additional training, outperforming both supervised and unsupervised defenses. Applying these interventions to the vision encoders of several state-of-the-art LVLMs yields substantial gains in visual question answering accuracy under typographic attack interference on RIO-Bench. arXiv: 2607.02494 (cs.CV, cs.CL), published 2026-07-02 by Bohan Liu, Wenqian Ye, Guangzhi Xiong, Zhenghao He, Sanchit Sinha, and Aidong Zhang.

Paper Overview

Research Area: NLP / Computer Vision Authors: Bohan Liu, Wenqian Ye, Guangzhi Xiong, Zhenghao He, Sanchit Sinha, Aidong Zhang Published: 2026-07-02 arXiv: 2607.02494 Categories: cs.CV, cs.CL

Abstract

Models trained via Contrastive Language-Image Pretraining (CLIP) serve as the foundational vision encoders for most modern Large Vision Language Models (LVLMs). Despite their widespread adoption, CLIP models exhibit a critical yet underexplored failure mode: irrelevant text appearing within images confounds visual representations, biasing them toward lexical meaning rather than true visual semantics. This robustness issue, commonly described as a Typographic Attack (TA), exposes a vulnerability that poses a significant risk to safety-critical applications such as autonomous driving.

To achieve interpretable and effective robustness against TA, the authors propose a novel, training-free mechanistic interpretability method. The approach:

Provides sampling-based interpretations of hidden state representations
Quantitatively attributes semantic vs. lexical attention to individual attention heads
Uses probabilistic analysis and circuit mining to isolate specific Vision Transformer (ViT) components that disproportionately encode lexical information, identifying the mechanical root of TA

Applying simple interventions (e.g., selectively adjusting attention weights) directly to the identified circuits significantly improves robustness against typographic attacks in targeted classification—without any additional training—and outperforms both supervised and unsupervised defense methods.

Results

When applied to the vision encoders of several state-of-the-art LVLMs, the proposed interventions yield significant improvements in visual question answering accuracy under typographic attack interference on RIO-Bench, confirming the effectiveness and generalization capability of the mechanistic approach.

---

*Auto-collected on 2026-07-06*