Visually Grounded Self-Reflection for Vision-Language Models via Reinforcement Learning

论文概要

研究领域: NLP 作者: Liyan Tang, Fangcong Yin, Greg Durrett 发布时间: 2026-07-02 arXiv: 2607.02490 分类: cs.CL, cs.CV

中文摘要

大型视觉-语言模型能够通过生成文本思维链（CoT）对多模态输入进行推理。CoT推理中展现的一项关键能力是自反思：重新审视先前的决策并纠正错误。然而，现有LVLMs在反思过程中往往未能正确关注视觉输入，限制了其将反馈转化为有依据的纠正的能力，尤其对分布外图像而言。为解决这一问题，我们提出一种新颖的强化学习训练框架VRRL，包含两个明确设计用于激发视觉基础自反思的组件。首先，我们在训练期间随机掩码轨迹前缀，以强调从不正确的中间预测中恢复，而非在早期犯错误。其次，我们引入来自经验回放缓冲区的缓冲roll-in，使模型接触多样的失败状态，必须学习纠正。我们在涉及表格和图表的视觉定位任务，以及空间导航基准上评估了我们的方法。虽然现成模型和传统微调模型在分布偏移下显著退化，但我们的方法通过有效利用自反思，相比标准RL和面向反思的微调基线，显著提升了平均分布外准确率。

原文摘要

Large vision-language models can reason over multimodal inputs by generating textual chains of thought (CoT). A key capability exhibited in CoT reasoning is self-reflection: revisiting earlier decisions and correcting previous errors. However, existing LVLMs often fail to properly attend to visual inputs during reflection, limiting their ability to translate feedback into grounded corrections, especially for out-of-distribution images. To address this issue, we propose a novel reinforcement learning training framework VRRL, with two components explicitly designed to elicit visually grounded self-reflection. First, we randomly mask trajectory prefixes during training to emphasize recovery from incorrect intermediate predictions rather than making early mistakes. Second, we introduce buffere...

--- *自动采集于 2026-07-06*

#论文 #arXiv #NLP #小凯