Paper Overview
Field: Computer Vision (CV) Authors: Dian Zheng, Harry Lee, Manyuan Zhang Published: 2025-06-13 arXiv: 2506.10669
Summary
Recent image generators have demonstrated impressive photorealism and instruction-following capabilities in single-image generation and editing. However, constrained by their architectures, they cannot achieve interleaved generation (text-image sequences), which has crucial applications in visual narratives, guidance, and embodied manipulation. Even the latest open-source Unified Multimodal Models (UMMs) exhibit limited performance in this regard.
This paper introduces InterleaveThinker — the first multi-agent pipeline designed to endow any existing image generator with interleaved generation capabilities:
- Planner agent: organizes the image-text input sequence and instructs the image generator on the required execution at each step.
- Critic agent: evaluates generator outputs, identifies samples that deviate from the planned instructions, and refines instructions for regeneration.
- InterleaveThinker improves performance across various image generators.
- On interleaved generation benchmarks, it achieves performance comparable to Nano Banana and GPT-5.
- Surprisingly, it also significantly enhances base-model reasoning: e.g., substantial gains on WISE and RISE with 4-step FLUX.2-klein.
Training Pipeline
1. SFT cold start: built on Interleave-Planner-SFT-80k and Interleave-Critic-SFT-112k. 2. Reinforcement learning: GRPO is used to develop Interleave-Critic-RL-13k, strengthening step-by-step instruction correction across generation trajectories. 3. Stepwise rewards: since a single interleaved trajectory can involve more than 25 generator calls, optimizing the entire trajectory is computationally impractical. The authors propose an accuracy reward and a stepwise reward so that single-step RL can effectively guide the whole generation trajectory.