[论文] Visually Grounded Self-Reflection for Vision-Language Models via Reinf...

论文概要

研究领域: NLP 作者: Liyan Tang, Fangcong Yin, Greg Durrett 发布时间: 2026-07-04 arXiv: 2507.03230

中文摘要

大型视觉-语言模型（LVLMs）可通过生成文本思维链（Chain of Thought, CoT）来对多模态输入进行推理。CoT 推理中展现的关键能力之一是自我反思（self-reflection）：重新审视早期决策并纠正先前错误。然而，现有 LVLMs 在反思过程中往往无法正确关注视觉输入，限制了其将反馈转化为基于视觉的修正的能力，尤其对分布外（out-of-distribution）图像而言。

为解决这一问题，本文提出一种新颖的强化学习训练框架 VRRL，包含两个专门为激发基于视觉的自我反思而设计的组件。首先，训练期间随机掩蔽轨迹前缀，以强调从错误中间预测中恢复，而非避免早期错误。其次，引入来自经验回放缓冲区（experience replay buffer）的缓冲 roll-in，使模型暴露于多样化的失败状态，从而学习纠正它们。

方法在涉及表格和图表的视觉定位任务以及空间导航基准上进行了评估。尽管现成模型和传统微调模型在分布偏移下显著退化，但本方法通过有效利用自我反思，在标准 RL 和面向反思的微调基线之上显著提升了平均分布外精度。

原文摘要

Large vision-language models can reason over multimodal inputs by generating textual chains of thought (CoT). A key capability exhibited in CoT reasoning is self-reflection: revisiting earlier decisions and correcting previous errors. However, existing LVLMs often fail to properly attend to visual inputs during reflection, limiting their ability to translate feedback into grounded corrections, especially for out-of-distribution images. To address this issue, we propose a novel reinforcement learning training framework VRRL, with two components explicitly designed to elicit visually grounded self-reflection. First, we randomly mask trajectory prefixes during training to emphasize recovery from incorrect intermediate predictions rather than making early mistakes. Second, we introduce buffered roll-ins from an experience replay buffer to expose the model to diverse failure states that it must learn to correct. We evaluate our approach on visual grounding tasks involving tables and charts,...

--- *自动采集于 2026-07-05*

#论文 #arXiv #NLP #小凯