[论文] Towards Robustness against Typographic Attack with Training-free Conce...

论文概要

研究领域: NLP 作者: Bohan Liu, Wenqian Ye, Guangzhi Xiong 发布时间: 2026-07-04 arXiv: 2507.03233

中文摘要

通过对比语言-图像预训练（CLIP）训练的模型是大多数现代大型视觉-语言模型（LVLMs）的基础视觉编码器。尽管被广泛采用，CLIP 模型却表现出一种关键但未被充分探索的失效模式：图像中出现的无关文本会混淆视觉表示，使其偏向词汇语义而非真正的视觉语义。这种鲁棒性问题通常被称为排版攻击（Typographic Attack, TA），暴露了安全关键型应用（如自动驾驶）中的重大风险。

为实现对 TA 的可解释且有效的鲁棒性，本文提出一种新颖的、无需训练的机械可解释性（mechanistic interpretability）方法。该方法提供对隐藏状态表示的基于采样的解释，并以定量方式将语义焦点与词汇焦点归因于单个注意力头（attention head）。通过概率分析和电路挖掘（circuit mining），研究分离出不成比例编码词汇信息的特定 Vision Transformer（ViT）组件，从而识别 TA 的机械根源。

研究进一步表明，对识别出的电路直接应用简单干预（无需任何额外训练）即可显著提升目标分类对排版攻击的鲁棒性。这些干预措施（如选择性调整注意力权重）优于监督式和免训练防御方法。实验表明，将所提干预应用于多个最先进 LVLM 的视觉编码器，在 RIO-Bench 的排版攻击干扰下显著提升了视觉问答（VQA）精度，证实了该机械方法的有效性和泛化性。

原文摘要

Models trained via Contrastive Language-Image Pretraining (CLIP) serve as the foundational vision encoders for most modern Large Vision Language Models (LVLMs). Despite their widespread adoption, CLIP models exhibit a critical yet underexplored failure mode: irrelevant text appearing within images confounds visual representations, biasing them toward lexical meaning rather than true visual semantics. This robustness issue, commonly described as a Typographic Attack (TA), exposes a vulnerability that poses a significant risk to safety-critical applications such as autonomous driving. To achieve interpretable and effective robustness against TA, we propose a novel, training-free mechanistic interpretability method. Our method provides sampling-based interpretations of hidden state representations and quantitatively attributes semantic versus lexical focus to individual attention heads. Through probabilistic analysis and circuit mining, we isolate specific Vision Transformer (ViT) compone...

--- *自动采集于 2026-07-05*

#论文 #arXiv #NLP #小凯