Paper Overview
Field: Computer Vision Authors: Shravan Venkatraman, Ritesh Thawkar, Omkar Thawkar Published: 2026-06-27 arXiv: 2606.27373
Summary
Recently, self-evolving large multimodal models (LMMs) have received attention for improving visual reasoning in a purely unsupervised setting. However, multi-role self-play and self-consistency reward schemes in existing self-evolving LMMs optimize answer agreement without ensuring the decoder attends to visual content, relying instead on statistical language priors to produce self-consistent outputs.
This leads to a persistent failure mode termed visual under-conditioning, where the decoder relies on language priors rather than the image during generation, manifesting as insufficient attention to visual tokens. As a result, current self-evolving LMMs struggle on vision-language understanding tasks such as image captioning and visual question answering.
Method: VISE (Visual Invariance Self-Evolution)
VISE is a purely unsupervised self-evolving framework that directly regularizes the model's visual conditioning policy through two complementary invariance rewards:
- Geometric invariance reward: enforces spatial consistency under known transformations.
- Semantic invariance reward: penalizes evidence-free generation by requiring the model to recognize the absence of evidence when predicted regions are perturbed.
- +16.85 CIDEr gain on COCO
- +19.66 CIDEr gain on TextCaps
- 5.0 ChairS-I point reduction in object hallucination
- Generalizes across four model families and scales
VISE operates within a single model, requiring no expert roles, external reward models, or annotations.
Results
Evaluated on 18 benchmarks using Qwen3-VL-2B as the base model: