Visually Grounded Self-Reflection for Vision-Language Models via Reinforcement Learning (VRRL)

Forum topic · 小凯 · 2026-07-06

Summary

This post introduces the arXiv paper 2607.02490 by Liyan Tang, Fangcong Yin, and Greg Durrett, proposing VRRL, a reinforcement learning training framework that teaches large vision-language models (LVLMs) to perform visually grounded self-reflection. While LVLMs can reason over multimodal inputs via textual chains of thought, they often fail to properly attend to visual inputs during reflection, limiting their ability to turn feedback into grounded corrections, especially on out-of-distribution images. VRRL addresses this with two components: random masking of trajectory prefixes during training to emphasize recovery from incorrect intermediate predictions, and buffer roll-ins from an experience replay buffer exposing the model to diverse failure states requiring correction. Evaluated on visually grounded tasks involving tables, charts, and spatial navigation benchmarks, VRRL significantly improves average out-of-distribution accuracy compared to standard RL and reflection-oriented fine-tuning baselines, which degrade substantially under distribution shift.

Paper Overview

Field: NLP Authors: Liyan Tang, Fangcong Yin, Greg Durrett Date: 2026-07-02 arXiv: 2607.02490 Categories: cs.CL, cs.CV

Summary

Large vision-language models can reason over multimodal inputs by generating textual chains of thought (CoT). A key capability exhibited in CoT reasoning is self-reflection: revisiting earlier decisions and correcting previous errors. However, existing LVLMs often fail to properly attend to visual inputs during reflection, limiting their ability to translate feedback into grounded corrections, especially for out-of-distribution images.

To address this issue, the authors propose a novel reinforcement learning training framework VRRL, with two components explicitly designed to elicit visually grounded self-reflection:

1. Random trajectory-prefix masking during training, to emphasize recovery from incorrect intermediate predictions rather than making early mistakes. 2. Buffer roll-ins from an experience replay buffer, exposing the model to diverse failure states it must learn to correct.

The method was evaluated on visually grounded tasks involving tables and charts, as well as spatial navigation benchmarks. While off-the-shelf models and conventionally fine-tuned models degrade significantly under distribution shift, VRRL leverages self-reflection effectively and substantially improves average out-of-distribution accuracy compared to standard RL and reflection-oriented fine-tuning baselines.

--- *Auto-collected on 2026-07-06*

Visually Grounded Self-Reflection for Vision-Language Models via Reinforcement Learning (VRRL)

Summary

Paper Overview

Summary

Tags