Towards Robustness against Typographic Attack with Training-free Mechanistic Interpretability in CLIP Models

Forum topic · 小凯 · 2026-07-05

Summary

This forum post introduces an arXiv paper (2507.03233) addressing the typographic attack (TA) vulnerability in CLIP-based vision encoders used by large vision-language models (LVLMs). CLIP models can be misled by irrelevant text appearing inside images, biasing visual representations toward lexical meaning instead of true visual semantics — a risk in safety-critical applications like autonomous driving. The authors propose a novel, training-free mechanistic interpretability method that provides sampling-based interpretations of hidden states and quantitatively attributes semantic versus lexical focus to individual attention heads. Using probabilistic analysis and circuit mining, they isolate specific Vision Transformer components that disproportionately encode lexical information, identifying the mechanical root of TA. Simple interventions on the identified circuits — such as selectively adjusting attention weights — require no additional training and outperform both supervised and training-free defenses. Applying these interventions to the vision encoders of multiple state-of-the-art LVLMs significantly improves visual question answering accuracy under typographic attacks on RIO-Bench, demonstrating the effectiveness and generalization of the mechanistic approach.

论文概要

研究领域: NLP 作者: Bohan Liu, Wenqian Ye, Guangzhi Xiong 发布时间: 2026-07-04 arXiv: 2507.03233

English Translation

Models trained via Contrastive Language-Image Pretraining (CLIP) serve as the foundational vision encoders for most modern Large Vision Language Models (LVLMs). Despite their widespread adoption, CLIP models exhibit a critical yet underexplored failure mode: irrelevant text appearing within images confounds visual representations, biasing them toward lexical meaning rather than true visual semantics. This robustness issue, commonly described as a Typographic Attack (TA), exposes a vulnerability that poses a significant risk to safety-critical applications such as autonomous driving.

To achieve interpretable and effective robustness against TA, the authors propose a novel, training-free mechanistic interpretability method. The method provides sampling-based interpretations of hidden state representations and quantitatively attributes semantic versus lexical focus to individual attention heads. Through probabilistic analysis and circuit mining, the study isolates specific Vision Transformer (ViT) components that disproportionately encode lexical information, thereby identifying the mechanical root of TA.

The study further shows that applying simple interventions directly on the identified circuits — without any additional training — significantly improves robustness of targeted classification against typographic attacks. These interventions, such as selectively adjusting attention weights, outperform both supervised and training-free defense methods. Experiments demonstrate that applying the proposed interventions to the vision encoders of multiple state-of-the-art LVLMs significantly improves visual question answering (VQA) accuracy under typographic attack perturbations on RIO-Bench, confirming the effectiveness and generalization of this mechanistic approach.

Key Contributions

A training-free mechanistic interpretability framework for analyzing hidden state representations in CLIP-style ViTs
Quantitative attribution of semantic vs. lexical focus to individual attention heads
Circuit mining that pinpoints the mechanical origin of typographic attack vulnerability
Simple, training-free circuit interventions that outperform existing defenses and generalize across LVLMs on RIO-Bench