Paying More Attention to Visual Tokens in Self-Evolving Large Multimodal Models

Forum topic · 小凯 · 2026-06-27

Summary

This paper addresses a failure mode in self-evolving large multimodal models (LMMs) that the authors term visual under-conditioning. Existing self-evolving LMMs rely on multi-role self-play and self-consistency reward schemes that optimize for answer agreement, but do not ensure the decoder actually attends to visual content; instead, models can rely on statistical language priors to produce self-consistent outputs. The authors propose introducing a visual-token attention mechanism that explicitly forces the decoder to focus on visual content during self-evolution, improving visual reasoning in a purely unsupervised setting. The work is in the computer vision domain, authored by Shravan Venkatraman, Ritesh Thawkar, and Omkar Thawakar, and is available on arXiv as 2606.27373.

Paper Overview

Field: Computer Vision (CV) Authors: Shravan Venkatraman, Ritesh Thawkar, Omkar Thawakar Published: 2026-06-27 arXiv: 2606.27373

Abstract

Recently, self-evolving large multimodal models (LMMs) have received attention for improving visual reasoning in a purely unsupervised setting. However, multi-role self-play and self-consistency reward schemes in existing self-evolving LMMs optimize answer agreement without ensuring the decoder attends to visual content, relying instead on statistical language priors to produce self-consistent outputs. This leads to a persistent failure mode the authors term visual under-conditioning, where the decoder under-weights visual inputs during generation.

Contribution

The paper introduces a visual-token attention mechanism that explicitly forces the decoder to attend to visual content during the self-evolution process, mitigating visual under-conditioning and improving visual reasoning without requiring supervised data.

---

*Source: forum post on zhichai.net, auto-collected 2026-06-27.*