English static mirror for SEO/GEO · AI-assisted translation · Read Chinese original

Towards Robustness against Typographic Attack with Training-free Mechanistic Interpretability

Forum topic · 小凯 · 2026-07-05

Summary

This paper (arXiv:2507.03233) addresses the Typographic Attack (TA) vulnerability in CLIP-based vision encoders, where irrelevant text within images biases visual representations toward lexical meaning rather than true visual semantics. The authors propose a novel, training-free mechanistic interpretability method that provides sampling-based interpretations of hidden state representations and quantitatively attributes semantic versus lexical focus to individual attention heads. Using probabilistic analysis and circuit mining, they isolate specific Vision Transformer (ViT) components that disproportionately encode lexical information, identifying the mechanistic root of TA. Simple interventions on these identified circuits—such as selectively adjusting attention weights—significantly improve classification robustness against typographic attacks without any additional training, outperforming both supervised and training-free defense baselines. Applied to the vision encoders of multiple state-of-the-art LVLMs, the interventions improve visual question answering accuracy under TA perturbations on RIO-Bench, demonstrating the effectiveness and generalization of the mechanistic approach for safety-critical applications such as autonomous driving.

Paper Overview

  • Field: NLP
  • Authors: Bohan Liu, Wenqian Ye, Guangzhi Xiong
  • arXiv: 2507.03233
  • Key Points

  • Models trained via Contrastive Language-Image Pretraining (CLIP) serve as the foundational vision encoders for most modern Large Vision Language Models (LVLMs), yet they exhibit a critical failure mode: irrelevant text appearing within images confounds visual representations, biasing them toward lexical meaning rather than true visual semantics. This robustness issue is known as a Typographic Attack (TA) and poses significant risk to safety-critical applications such as autonomous driving.
  • The authors propose a novel, training-free mechanistic interpretability method. It provides sampling-based interpretations of hidden state representations and quantitatively attributes semantic versus lexical focus to individual attention heads.
  • Through probabilistic analysis and circuit mining, the study isolates specific Vision Transformer (ViT) components that disproportionately encode lexical information, identifying the mechanistic root cause of TA.
  • Simple interventions applied directly to the identified circuits—such as selectively adjusting attention weights—require no additional training and significantly improve classification robustness against typographic attacks. These interventions outperform both supervised and training-free defense methods.
  • When applied to the vision encoders of multiple state-of-the-art LVLMs, the proposed interventions substantially improve visual question answering (VQA) accuracy under TA perturbations on RIO-Bench, confirming the effectiveness and generalization of this mechanistic approach.

Original Abstract (excerpt)

> Models trained via Contrastive Language-Image Pretraining (CLIP) serve as the foundational vision encoders for most modern Large Vision Language Models (LVLMs). Despite their widespread adoption, CLIP models exhibit a critical yet underexplored failure mode: irrelevant text appearing within images confounds visual representations, biasing them toward lexical meaning rather than true visual semantics. This robustness issue, commonly described as a Typographic Attack (TA), exposes a vulnerability that poses a significant risk to safety-critical applications such as autonomous driving.

---

*Auto-collected on 2026-07-05*

Tags

#typographic-attack#clip#mechanistic-interpretability#vision-language-models#adversarial-robustness#attention-heads#vqa#arxiv

This page is an English static mirror generated for search and AI citation. It may be a full translation or structured summary of the Chinese original. Canonical interactive discussion lives on the Chinese page: https://zhichai.net/topic/178208433