UMM-Reflection: Learning Native Reflection in Unified Multimodal Models with Reinforcement Learning

Forum topic · 小凯 · 2026-09-30

Summary

UMM-Reflection is a reinforcement learning framework that trains unified multimodal models to reflect on and repair their own image generations. Because a revision's usefulness is only known after rendering, reflection text and image generation must be learned jointly across the full loop. The method applies RL to complete reflection trajectories inside one unified model: sibling trajectories share the same initial image so group-relative advantage can compare reflection strategies, and trajectory-level advantages jointly update both reflection tokens and flow-based modifications, avoiding per-turn credit assignment. Unlike single-turn editing or pipelines with external critics, credit flows across turns and to both roles of the same model, requiring no verifier at inference. On BAGEL, UMM-Reflection improves GenEval by 12.05 points over SFT, with gains transferring to unseen benchmarks: WISE (+10.97), OneIG-Bench (+3.48), and T2I-CompBench++ (+4.63). Paper: arXiv 2609.35767.

Paper Overview

Field: Computer Vision (CV) Authors: Yijia Fan, Ziqi Huang, Zhongang Cai, Yan Li, Zimo Wen, Wanqi Yin, Haiwen Diao, Ziwei Liu Published: 2026-09-28 arXiv: 2609.35767

Abstract

Unified multimodal models can both look at and render images, so in principle they can repair their own generations: diagnose what an image gets wrong, revise it, observe the result, and diagnose again. Whether a revision helps is known only after it is rendered, so the reflection text and the image generation must be learned jointly, over the whole loop. Supervised fine-tuning (SFT) on reflection trajectories gives a cold start but does not find the high-success repair paths, and naive RL that optimizes only the renderer or only one head leaves most of the gain untapped.

UMM-Reflection applies reinforcement learning (RL) to complete reflection trajectories inside one unified model:

Group-relative advantage: sibling trajectories share one initial image, so different reflection strategies can be compared directly.
Trajectory-level advantage: updates both reflection tokens and flow-based modifications simultaneously, avoiding the combinatorial explosion of per-turn credit assignment.
Cross-turn, cross-role credit: unlike single-turn editing or pipelines with external critics, credit flows across turns and to both roles of the same model; no verifier is needed at inference.

Results

On BAGEL, UMM-Reflection improves GenEval by 12.05 points over SFT, and the gains transfer to benchmarks not used in training:

| Benchmark | Improvement | |---|---| | WISE | +10.97 | | OneIG-Bench | +3.48 | | T2I-CompBench++ | +4.63 |