Images Talk Back: Guiding MMDiTs by Suppressing Image-to-Text Feedback

Forum topic · 小凯 · 2026-10-13

Summary

This paper investigates how multimodal diffusion transformers (MMDiTs) handle image-to-text (I2T) feedback, where the evolving image updates internal text representations via joint attention, which then guide subsequent image updates. The authors find that this feedback tends to preserve objects and attributes already present in the image—including errors—rather than correcting deviations from the prompt. Transplanting I2T messages can rescue failed generations or disrupt successful ones, and at matched intervention strengths, suppressing I2T feedback improves generation while opposite controls degrade it. Based on these findings, they propose Semantic Feedback Guidance (SFG), which suppresses I2T feedback by guiding away from an auxiliary branch that strengthens it and weakens text-to-image conditioning, plus an in-place variant (iSFG) that requires no extra model evaluations. SFG only needs to run during roughly the first 20% of sampling steps, costing about 1.2x base NFE without classifier-free guidance. Experiments across three image generation backbones show improved alignment and preference scores: on FLUX, SFG stacked with CFG raises GenEval from 0.6661 to 0.7479; on SD3.5M, iSFG adds +1.33 HPSv2.1 to CFG with zero extra NFE. The method also transfers to text-to-video and image-to-video generation, improving VBench scores. Paper: arXiv:2610.13941.

Paper Overview

Field: Computer Vision Authors: Yucheng Yang, Xiaoxiao Sun, Shusheng Yang, Jiakang Chen, Serena Yeung-Levy Published: 2026-10-09 arXiv: 2610.13941

Introduction

Multimodal diffusion transformers (MMDiTs) allow the evolving image to update internal text representations through joint attention. These representations then guide subsequent image updates, raising a question: does this feedback help correct deviations from the prompt, or preserve them?

Key Findings

Image-to-text (I2T) feedback preserves the objects and attributes already represented in the evolving image, including errors.
Transplanting I2T messages can rescue failure cases or disrupt successful ones.
At matched intervention strengths, suppressing I2T feedback improves generation, whereas opposite controls degrade it.

Method: Semantic Feedback Guidance (SFG)

These findings motivate Semantic Feedback Guidance (SFG), which suppresses the influence of I2T feedback by guiding away from an auxiliary branch that strengthens I2T feedback and weakens text-to-image conditioning.

An in-place variant, iSFG, directly suppresses I2T feedback while reinforcing text-to-image conditioning, requiring no additional model evaluations.

Temporal intervention studies show that running the SFG branch during only the first ~20% of image sampling steps is sufficient, at roughly 1.2x base NFE (without CFG).

Results

Experiments across three image generation backbones show consistent improvements in alignment and human-preference scores.
FLUX: SFG stacked on CFG improves GenEval from 0.6661 to 0.7479.
SD3.5M: iSFG adds +1.33 HPSv2.1 on top of CFG with zero additional NFE.
The method transfers to text-to-video and image-to-video generation, improving VBench scores.