[论文] Paying More Attention to Visual Tokens in Self-Evolving Large Multimodal Models

论文概要

研究领域: 计算机视觉 作者: Shravan Venkatraman, Ritesh Thawkar, Omkar Thawkar 发布时间: 2026-06-27 arXiv: 2606.27373

中文摘要

最近，自进化大型多模态模型（LMMs）在纯无监督设置中改善视觉推理方面受到关注。然而，现有自进化LMMs中的多角色自博弈和自一致性奖励方案优化答案一致性，却不确保解码器关注视觉内容，而是依赖统计语言先验来产生自一致的输出。这导致了一种持久的失败模式，我们称之为视觉欠条件化（visual under-conditioning），即解码器在生成过程中依赖语言先验而非图像，表现为对视觉令牌关注不足。为此，我们提出VISE（视觉不变性自进化），一种纯无监督自进化框架，通过两种互补的不变性奖励直接正则化模型的视觉条件化策略：几何不变性奖励在已知变换下强制执行空间一致性；语义不变性奖励通过要求模型在预测区域被扰动时识别证据缺失，惩罚无证据生成。VISE在单一模型内运行，无需专家角色、外部奖励模型或标注。在18个基准测试上，使用Qwen3-VL-2B作为基础模型，VISE在COCO上实现+16.85 CIDEr增益，在TextCaps上实现+19.66 CIDEr增益，将物体幻觉减少5.0 Chair-I点，并在四个模型家族和规模上泛化。

原文摘要

Recently, self-evolving large multimodal models (LMMs) have received attention for improving visual reasoning in a purely unsupervised setting. However, multi-role self-play and self-consistency reward schemes in existing self-evolving LMMs optimize answer agreement without ensuring the decoder attends to visual content, relying instead on statistical language priors to produce self consistent outputs. This leads to a persistent failure mode we term visual under-conditioning, where the decoder relies on language priors rather than the image during generation, manifesting as insufficient attention to visual tokens. As a result, current self-evolving LMMs struggle on vision-language understanding tasks such as image captioning and visual question answering. To address this, we propose VISE...

--- *自动采集于 2026-06-27*

#论文 #arXiv #cs.CV #小凯