RepFusion: Leveraging Multimodal LLM Priors for Denoising in Text-to-Image Generation

Forum topic · 小凯 · 2026-06-16

Summary

RepFusion (arXiv:2606.14700) is a computer vision paper by Xichen Pan, Aashu Singh, and Satya Narayan Shukla that rethinks how large language models are used in text-to-image (T2I) systems. While traditional T2I pipelines use LLMs only for text encoding and rely on newly trained generative backbones for denoising, RepFusion repurposes the multimodal LLM (MLLM) itself as an encoder of noisy visual representations. Building on representation autoencoders (RAEs), which produce semantically structured latent spaces compatible with pretrained LLM priors, the authors use MLLM outputs as the conditioning signal for a diffusion transformer. In controlled comparisons under similar inference budgets, RepFusion outperforms baselines that allocate equivalent capacity to newly initialized denoisers. The results indicate that MLLMs provide strong priors for visual representation denoising and that repeated MLLM conditioning at test time can be an effective use of test-time compute in modern T2I systems.

Paper Overview

Field: Computer Vision (CV)
Authors: Xichen Pan, Aashu Singh, Satya Narayan Shukla
Published: 2026-06-12
arXiv: 2606.14700

Abstract

Large language models (LLMs) are widely used in text-to-image (T2I) systems, but they are typically limited to text encoding, while denoising is handled by newly trained generative backbones. The emergence of representation autoencoders (RAEs) shifts the generation target toward semantically structured visual representations, creating a latent space that is more compatible with pretrained LLM priors.

Inspired by multimodal LLMs (MLLMs), where an MLP projector is sufficient to align clean visual representations with a pretrained LLM, the authors repurpose the MLLM itself as a noisy representation encoder, extending this mechanism from clean to noisy inputs. They present RepFusion, which uses the resulting MLLM outputs as the conditioning signal for a diffusion transformer.

Key Findings

In controlled comparisons at similar inference budgets, RepFusion outperforms baselines that allocate comparable capacity to newly initialized denoisers.
MLLMs provide strong priors for visual representation denoising.
Repeated MLLM conditioning at test time is an effective way to leverage test-time compute in modern T2I systems.

---

*Auto-collected on 2026-06-16.*