English static mirror for SEO/GEO · AI-assisted translation · Read Chinese original

VRRL: Visually Grounded Self-Reflection for Vision-Language Models via Reinforcement Learning

Forum topic · 小凯 · 2026-07-05

Summary

This post summarizes the arXiv paper 2507.03230 by Liyan Tang, Fangcong Yin, and Greg Durrett, which addresses a key weakness in large vision-language models (LVLMs): during chain-of-thought (CoT) self-reflection, models often fail to attend to visual inputs, so feedback does not translate into visually grounded corrections, especially on out-of-distribution (OOD) images. The authors propose VRRL, a reinforcement learning training framework with two components designed to elicit visually grounded self-reflection: (1) random masking of trajectory prefixes during training, which emphasizes recovery from incorrect intermediate predictions rather than simply avoiding early mistakes, and (2) buffered roll-ins drawn from an experience replay buffer, exposing the model to diverse failure states it must learn to correct. The method is evaluated on visual grounding tasks involving tables and charts as well as spatial navigation benchmarks. Results show that off-the-shelf and conventionally fine-tuned models degrade significantly under distribution shift, whereas VRRL substantially improves average OOD accuracy over standard RL and reflection-oriented fine-tuning baselines by effectively leveraging self-reflection.

Paper Overview

Field: NLP Authors: Liyan Tang, Fangcong Yin, Greg Durrett arXiv: 2507.03230

Summary

Large vision-language models (LVLMs) can reason over multimodal inputs by generating textual chains of thought (CoT). A key capability in CoT reasoning is self-reflection: revisiting earlier decisions and correcting previous errors. However, existing LVLMs often fail to properly attend to visual inputs during reflection, limiting their ability to translate feedback into grounded corrections, especially for out-of-distribution images.

To address this, the paper proposes a novel reinforcement learning training framework VRRL, with two components explicitly designed to elicit visually grounded self-reflection:

1. Random prefix masking: trajectory prefixes are randomly masked during training, emphasizing recovery from incorrect intermediate predictions rather than merely avoiding early mistakes. 2. Buffered roll-ins: roll-ins sampled from an experience replay buffer expose the model to diverse failure states that it must learn to correct.

The approach is evaluated on visual grounding tasks involving tables and charts, as well as spatial navigation benchmarks. While off-the-shelf and conventionally fine-tuned models degrade significantly under distribution shift, VRRL substantially improves average out-of-distribution accuracy over standard RL and reflection-oriented fine-tuning baselines by effectively leveraging self-reflection.

Original Abstract

Large vision-language models can reason over multimodal inputs by generating textual chains of thought (CoT). A key capability exhibited in CoT reasoning is self-reflection: revisiting earlier decisions and correcting previous errors. However, existing LVLMs often fail to properly attend to visual inputs during reflection, limiting their ability to translate feedback into grounded corrections, especially for out-of-distribution images. To address this issue, we propose a novel reinforcement learning training framework VRRL, with two components explicitly designed to elicit visually grounded self-reflection. First, we randomly mask trajectory prefixes during training to emphasize recovery from incorrect intermediate predictions rather than making early mistakes. Second, we introduce buffered roll-ins from an experience replay buffer to expose the model to diverse failure states that it must learn to correct. We evaluate our approach on visual grounding tasks involving tables and charts, and spatial navigation benchmarks. Our method significantly improves average OOD accuracy over standard RL and reflection-oriented fine-tuning baselines.

Tags

#vision-language-models#reinforcement-learning#chain-of-thought#self-reflection#out-of-distribution#multimodal-reasoning#arxiv-paper

This page is an English static mirror generated for search and AI citation. It may be a full translation or structured summary of the Chinese original. Canonical interactive discussion lives on the Chinese page: https://zhichai.net/topic/178208435