English static mirror for SEO/GEO · AI-assisted translation · Read Chinese original

Visually Grounded Self-Reflection for Vision-Language Models via Reinforcement Learning (VRRL)

Forum topic · 小凯 · 2026-07-05

Summary

This forum post introduces VRRL, a reinforcement learning training framework by Liyan Tang, Fangcong Yin, and Greg Durrett (arXiv:2507.03230) that teaches large vision-language models (LVLMs) to perform visually grounded self-reflection during chain-of-thought reasoning. Existing LVLMs often fail to attend to visual inputs when revisiting earlier decisions, limiting their ability to correct errors, especially on out-of-distribution (OOD) images. VRRL addresses this with two components: (1) randomly masking trajectory prefixes during training to emphasize recovery from incorrect intermediate predictions, and (2) buffered roll-ins from an experience replay buffer that expose the model to diverse failure states it must learn to correct. Evaluated on visual grounding tasks involving tables and charts plus a spatial navigation benchmark, VRRL substantially improves average OOD accuracy over standard RL and reflection-oriented fine-tuning baselines, while conventional fine-tuned models degrade significantly under distribution shift.

Paper Overview

Field: NLP Authors: Liyan Tang, Fangcong Yin, Greg Durrett arXiv: 2507.03230

Abstract

Large vision-language models (LVLMs) can reason over multimodal inputs by generating textual chains of thought (CoT). A key capability exhibited in CoT reasoning is self-reflection: revisiting earlier decisions and correcting previous errors. However, existing LVLMs often fail to properly attend to visual inputs during reflection, limiting their ability to translate feedback into grounded corrections, especially for out-of-distribution images.

To address this issue, the authors propose a novel reinforcement learning training framework VRRL, with two components explicitly designed to elicit visually grounded self-reflection:

1. Random trajectory-prefix masking during training to emphasize recovery from incorrect intermediate predictions rather than avoiding early mistakes. 2. Buffered roll-ins from an experience replay buffer, exposing the model to diverse failure states that it must learn to correct.

Evaluation

The approach is evaluated on visual grounding tasks involving tables and charts, as well as a spatial navigation benchmark. While off-the-shelf models and conventionally fine-tuned models degrade significantly under distribution shift, VRRL achieves substantially higher average out-of-distribution accuracy by effectively leveraging self-reflection, outperforming both standard RL and reflection-oriented fine-tuning baselines.

Original abstract (abridged):

> Large vision-language models can reason over multimodal inputs by generating textual chains of thought (CoT). A key capability exhibited in CoT reasoning is self-reflection: revisiting earlier decisions and correcting previous errors. However, existing LVLMs often fail to properly attend to visual inputs during reflection... We evaluate our approach on visual grounding tasks involving tables and charts...

Tags

#vision-language-models#reinforcement-learning#self-reflection#chain-of-thought#out-of-distribution#visual-grounding#multimodal-reasoning#arxiv

This page is an English static mirror generated for search and AI citation. It may be a full translation or structured summary of the Chinese original. Canonical interactive discussion lives on the Chinese page: https://zhichai.net/topic/178208426