Paper: Paying More Attention to Visual Tokens in Self-Evolving Large Multimodal Models

Summary

This post summarizes an arXiv paper (2606.27373) on improving self-evolving large multimodal models (LMMs). Existing self-evolving LMMs use multi-role self-play and self-consistency reward schemes to improve visual reasoning in a purely unsupervised setting. However, these schemes optimize answer agreement without ensuring the decoder actually attends to visual content, so the model can rely on statistical language priors to produce self-consistent outputs. The authors identify this failure mode as visual under-conditioning, where visual reasoning is driven by linguistic patterns rather than visual information. The paper proposes a visual token attention mechanism that explicitly forces the decoder to attend to visual content during self-evolution, ensuring reasoning is genuinely grounded in visual inputs. Field: computer vision.