Training-free Mechanistic Interpretability Defense Against Typographic Attacks in CLIP Models

Forum topic · 小凯 · 2026-07-05

Summary

A new paper (arXiv:2507.03233) by Bohan Liu, Wenqian Ye, and Guangzhi Xiong addresses typographic attacks (TA) in CLIP-based vision encoders, where irrelevant text embedded in images biases visual representations toward lexical meaning instead of true visual semantics. This vulnerability poses risks for safety-critical systems such as autonomous driving. The authors propose a novel, training-free mechanistic interpretability method that provides sampling-based interpretations of hidden state representations and quantitatively attributes semantic versus lexical focus to individual attention heads. Using probabilistic analysis and circuit mining, they isolate specific Vision Transformer (ViT) components that disproportionately encode lexical information, identifying the mechanical root of typographic attacks. Simple interventions on these circuits, such as selectively adjusting attention weights, significantly improve robustness to TA without any additional training, outperforming both supervised and training-free defenses. When applied to the vision encoders of multiple state-of-the-art large vision-language models, the interventions substantially improve visual question answering accuracy under TA perturbations on RIO-Bench, demonstrating the method's effectiveness and generalizability.

Paper Overview

Field: NLP
Authors: Bohan Liu, Wenqian Ye, Guangzhi Xiong
arXiv: 2507.03233

Key Points

Problem: Models trained via Contrastive Language-Image Pretraining (CLIP) are the foundational vision encoders for most modern Large Vision Language Models (LVLMs). However, they exhibit a critical failure mode: irrelevant text appearing within images confounds visual representations, biasing them toward lexical meaning rather than true visual semantics. This robustness issue is known as a Typographic Attack (TA) and poses significant risk to safety-critical applications such as autonomous driving.
Approach: The authors propose a novel, training-free mechanistic interpretability method to achieve interpretable and effective robustness against TA. The method:
Provides sampling-based interpretations of hidden state representations.
Quantitatively attributes semantic versus lexical focus to individual attention heads.
Uses probabilistic analysis and circuit mining to isolate specific Vision Transformer (ViT) components that disproportionately encode lexical information, revealing the mechanical root of TA.
Defense: Applying simple interventions directly to the identified circuits — such as selectively adjusting attention weights — requires no additional training and significantly improves robustness against typographic attacks.
Results:
The proposed interventions outperform both supervised and training-free defense baselines.
Applied to the vision encoders of multiple state-of-the-art LVLMs, they substantially improve visual question answering (VQA) accuracy under TA perturbations on RIO-Bench, confirming the effectiveness and generalizability of the mechanistic approach.

Original Abstract (excerpt)

> Models trained via Contrastive Language-Image Pretraining (CLIP) serve as the foundational vision encoders for most modern Large Vision Language Models (LVLMs). Despite their widespread adoption, CLIP models exhibit a critical yet underexplored failure mode: irrelevant text appearing within images confounds visual representations, biasing them toward lexical meaning rather than true visual semantics. This robustness issue, commonly described as a Typographic Attack (TA), exposes a vulnerability that poses a significant risk to safety-critical applications such as autonomous driving. To achieve interpretable and effective robustness against TA, we propose a novel, training-free mechanistic interpretability method...

*Automatically collected on 2026-07-05.*