[论文] Ask, Solve, Generate: Self-Evolving Unified Multimodal Understanding and Generation via Self-Consistency Rewards

论文概要

研究领域: 计算机视觉
作者: Ritesh Thawkar, Shravan Venkatraman, Omkar Thawkar
发布时间: 2026-06-27
arXiv: 2606.27376

中文摘要

大多数支持视觉理解和图像生成的统一大型多模态模型（LMM）仍依赖精心策划的后训练监督，如人工标注、偏好标签或外部奖励模型。我们探讨：一个统一的LMM能否仅使用未标注图像自主提升两种能力？我们提出自进化训练框架，包含三个内部角色：生成视觉问题的提议者（Proposer）、回答和评估问题的求解者（Solver），以及合成图像的生成器（Generator）。训练仅使用自推导的一致性信号，无需人工标注或外部奖励模型。我们引入求解器令牌熵（STE），一种基于令牌级预测不确定性的连续难度信号。对于图像生成，我们设计了多尺度内部评估方案，结合问答保真度评分与循环一致性描述。该框架在BLIP3o、BAGEL和VARGPT-v1.1架构上保持相同的角色分解和奖励逻辑。在八个理解指标上，我们的方法始终优于基础模型。在BAGEL上，MMMU实现+3.5%绝对增益，GenEval图像生成性能从82%提升至85%。

原文摘要

Most unified large multimodal models (LMMs) that support both visual understanding and image generation still rely on curated post-training supervision, such as human annotations, preference labels, or external reward models. We ask whether a unified LMM can improve both abilities autonomously using only unlabeled images. We propose a self-evolving training framework with three internal roles: a Proposer that generates visual questions, a Solver that answers and evaluates them, and a Generator that synthesizes images. Training uses only self-derived consistency signals, without human annotations, preference labels, or task-trained external reward/judge models. To stabilize learning, we introduce Solver Token Entropy (STE), a continuous difficulty signal based on token-level prediction...

自动采集于 2026-06-27

#论文 #arXiv #cs.CV #小凯