Paying More Attention to Visual Tokens in Self-Evolving Large Multimodal Models

Forum topic · 小凯 · 2026-06-27

Summary

This arXiv paper (2606.27373) addresses a failure mode in self-evolving large multimodal models (LMMs). Existing approaches rely on multi-role self-play and self-consistency reward schemes that optimize answer agreement without ensuring the decoder actually attends to visual content, causing the model to lean on statistical language priors to produce self-consistent outputs. The authors identify and name this persistent failure mode visual under-conditioning. Their proposed solution introduces a visual token attention mechanism that explicitly forces the decoder to focus on visual content during the self-evolution process, improving visual reasoning in a purely unsupervised setting. The work targets the computer vision research community and offers a targeted fix for grounding self-evolving multimodal models in visual inputs rather than language priors.

Paper Overview

Field: Computer Vision (CV) Authors: Shravan Venkatraman, Ritesh Thawkar, Omkar Thawakar Published: 2026-06-27 arXiv: 2606.27373

Abstract

Recently, self-evolving large multimodal models (LMMs) have received attention for improving visual reasoning in a purely unsupervised setting. However, multi-role self-play and self-consistency reward schemes in existing self-evolving LMMs optimize answer agreement without ensuring the decoder attends to visual content, relying instead on statistical language priors to produce self consistent outputs. This leads to a persistent failure mode we term visual under-conditioning, where the decoder relies insufficiently on visual inputs.

Key Contribution

The authors introduce a visual token attention mechanism that explicitly enforces the decoder to attend to visual content during the self-evolving process, addressing visual under-conditioning without requiring supervised data.

--- *Auto-collected on 2026-06-27*