Paying More Attention to Visual Tokens in Self-Evolving Large Multimodal Models

Forum topic · 小凯 · 2026-06-27

Summary

This paper addresses a failure mode in self-evolving large multimodal models (LMMs) called visual under-conditioning. Existing approaches rely on multi-role self-play and self-consistency reward schemes that optimize answer agreement in purely unsupervised settings, but they do not ensure the decoder actually attends to visual content, instead leveraging statistical language priors to produce self-consistent outputs. The authors propose a visual token attention mechanism that explicitly forces the decoder to attend to visual content during the self-evolution process. By increasing attention to visual tokens, the method aims to ground model reasoning in image evidence rather than language priors, improving visual reasoning performance without supervised labels. The work targets the computer vision research community and contributes to ongoing efforts on unsupervised self-improvement of multimodal models. Full details are available on arXiv (2606.27373).

Paper Overview

Research Area: Computer Vision (CV)

Authors: Shravan Venkatraman, Ritesh Thawkar, Omkar Thawakar

Published: 2026-06-27

arXiv: 2606.27373

Abstract

Recently, self-evolving large multimodal models (LMMs) have received attention for improving visual reasoning in a purely unsupervised setting. However, multi-role self-play and self-consistency reward schemes in existing self-evolving LMMs optimize answer agreement without ensuring the decoder attends to visual content, relying instead on statistical language priors to produce self-consistent outputs. This leads to a persistent failure mode the authors term visual under-conditioning, where the decoder under-weights visual inputs when generating answers.

Key Idea

The authors introduce a visual token attention mechanism that explicitly forces the decoder to attend to visual content during self-evolution, grounding the model's reasoning in image evidence rather than language priors.

Summary Points

Self-evolving LMMs can improve visual reasoning without supervision, but self-consistency rewards alone do not guarantee visual grounding.
Visual under-conditioning: models may produce self-consistent answers driven by language priors while ignoring visual input.
Proposed fix: explicitly increase/enforce attention on visual tokens during the self-evolving training process.

---

*Auto-collected on 2026-06-27. Translated and summarized from the original forum post.*