VRRL: Visually Grounded Self-Reflection for Vision-Language Models via Reinforcement Learning

Forum topic · 小凯 · 2026-07-05

Summary

Researchers Liyan Tang, Fangcong Yin, and Greg Durrett propose VRRL, a reinforcement learning training framework that teaches large vision-language models (LVLMs) to perform visually grounded self-reflection during chain-of-thought reasoning. While LVLMs can reason over multimodal inputs, existing models often fail to attend to visual inputs when reflecting on earlier decisions, limiting their ability to correct errors, especially on out-of-distribution (OOD) images. VRRL addresses this with two components: random masking of trajectory prefixes during training to emphasize recovery from incorrect intermediate predictions, and buffered roll-ins from an experience replay buffer to expose the model to diverse failure states it must learn to fix. Evaluated on visual grounding tasks involving tables and charts as well as spatial navigation benchmarks, VRRL significantly outperforms standard RL and reflection-oriented fine-tuning baselines, with substantially higher average out-of-distribution accuracy. The paper is available on arXiv as 2507.03230.

Paper Overview

Field: NLP Authors: Liyan Tang, Fangcong Yin, Greg Durrett arXiv: 2507.03230

Abstract

Large vision-language models can reason over multimodal inputs by generating textual chains of thought (CoT). A key capability exhibited in CoT reasoning is self-reflection: revisiting earlier decisions and correcting previous errors. However, existing LVLMs often fail to properly attend to visual inputs during reflection, limiting their ability to translate feedback into grounded corrections, especially for out-of-distribution images.

To address this issue, the authors propose a novel reinforcement learning training framework, VRRL, with two components explicitly designed to elicit visually grounded self-reflection:

1. Random prefix masking: trajectory prefixes are randomly masked during training to emphasize recovery from incorrect intermediate predictions rather than avoiding early mistakes. 2. Buffered roll-ins: roll-ins from an experience replay buffer expose the model to diverse failure states that it must learn to correct.

Evaluation

The approach is evaluated on visual grounding tasks involving tables and charts, as well as on spatial navigation benchmarks. While off-the-shelf models and conventionally fine-tuned models degrade significantly under distribution shift, VRRL effectively leverages self-reflection and substantially improves average out-of-distribution accuracy over both standard RL and reflection-oriented fine-tuning baselines.

Paper link: https://arxiv.org/abs/2507.03230