Ask, Solve, Generate: Self-Evolving Unified Multimodal Understanding and Generation via Self-Consistency Rewards

Forum topic · 小凯 · 2026-06-27

Summary

This paper introduces a self-evolving training framework that lets unified large multimodal models (LMMs) improve both visual understanding and image generation using only unlabeled images. The framework assigns three internal roles: a Proposer that generates visual questions, a Solver that answers and evaluates them, and a Generator that synthesizes images. Training relies solely on self-derived consistency signals, requiring no human annotations, preference labels, or external reward models. The authors also propose Solver Token Entropy (STE), a continuous difficulty signal based on token-level prediction uncertainty, and a multi-scale internal evaluation scheme for image generation that combines question-answering fidelity scoring with cycle-consistency captioning. Applied to BLIP3o, BAGEL, and VARGPT-v1.1 architectures, the method consistently outperforms base models across eight understanding benchmarks. On BAGEL, it achieves a +3.5% absolute gain on MMMU and improves GenEval image generation performance from 82% to 85%.

Paper Overview

Field: Computer Vision Authors: Ritesh Thawkar, Shravan Venkatraman, Omkar Thawkar arXiv: 2606.27376

Abstract

Most unified large multimodal models (LMMs) that support both visual understanding and image generation still rely on curated post-training supervision, such as human annotations, preference labels, or external reward models. This work asks whether a unified LMM can improve both abilities autonomously using only unlabeled images.

The proposed self-evolving training framework uses three internal roles:

Proposer — generates visual questions
Solver — answers and evaluates questions
Generator — synthesizes images

Training uses only self-derived consistency signals, without human annotations, preference labels, or task-trained external reward/judge models.

Key Contributions

Solver Token Entropy (STE): a continuous difficulty signal based on token-level prediction uncertainty, used to stabilize learning.
Multi-scale internal evaluation for image generation: combines question-answering fidelity scoring with cycle-consistency captioning.

Results

The framework was applied to BLIP3o, BAGEL, and VARGPT-v1.1 architectures while preserving the same role decomposition and reward logic:

Consistently outperforms base models on eight understanding benchmarks.
On BAGEL: +3.5% absolute gain on MMMU, and GenEval image generation performance improves from 82% to 85%.

--- *Auto-collected on 2026-06-27*