RepFusion: Leveraging Multimodal LLM Priors for Denoising in Representation Space for Text-to-Image Generation

Forum topic · 小凯 · 2026-06-16

Summary

RepFusion (arXiv:2606.14700) is a computer vision paper by Xichen Pan, Aashu Singh, and Satya Narayan Shukla that repurposes multimodal large language models (MLLMs) as denoising encoders for text-to-image generation. Traditional T2I systems use LLMs only for text encoding while a newly trained generative backbone handles denoising. The authors build on representation autoencoders (RAEs), which shift the generation target toward semantically structured visual representations more compatible with pretrained LLM priors. Inspired by MLLMs—where an MLP projector suffices to align clean visual representations with a pretrained LLM—RepFusion extends this mechanism to noisy inputs, using MLLM outputs as the conditioning signal for a diffusion transformer. In controlled comparisons at similar inference budgets, RepFusion outperforms baselines that allocate comparable capacity to newly initialized denoisers. The results suggest MLLMs provide strong priors for denoising visual representations, and that conditioning on evolving noisy representations allows test-time compute to be effectively reused across the repeated MLLM conditioning steps of modern T2I systems.

Paper Overview

Research area: Computer Vision
Authors: Xichen Pan, Aashu Singh, Satya Narayan Shukla
Published: 2026-06-12
arXiv: 2606.14700

Abstract

Large language models (LLMs) are widely used in text-to-image (T2I) systems, but they are typically limited to text encoding, while denoising is handled by newly trained generative backbones. The emergence of representation autoencoders (RAEs) shifts the generation target toward semantically structured visual representations, creating a latent space that is more compatible with pretrained LLM priors.

Inspired by multimodal LLMs (MLLMs), where an MLP projector is sufficient to align clean visual representations with a pretrained LLM, the authors repurpose the MLLM itself as a noisy representation encoder, extending this mechanism from clean to noisy inputs. They present RepFusion, which uses the resulting MLLM outputs as the conditioning signal for a diffusion transformer.

Key Findings

In controlled comparisons at similar inference budgets, RepFusion outperforms baselines that allocate equivalent capacity to newly initialized denoisers.
MLLMs provide strong priors for denoising semantically structured visual representations.
Conditioning on evolving noisy representations enables test-time compute to be effectively reused for the repeated MLLM conditioning required in modern T2I systems.

Discussion

This work points toward a shift in T2I architecture: rather than training dedicated generative backbones from scratch, pretrained multimodal LLMs could serve as reusable denoising priors, potentially reducing training cost while leveraging the semantic knowledge already embedded in large models. Reader discussion on the implications for next-generation T2I pipelines is welcome below.

---

*Auto-collected on 2026-06-16*