Representation Forcing: Bottleneck-Free Unified Multimodal Models Without External VAEs

Forum topic · 小凯 · 2026-06-02

Summary

This post introduces Representation Forcing (RF), a method for unified multimodal models (UMMs) that removes the structural bottleneck caused by frozen, separately pretrained VAEs. Existing UMMs typically rely on such VAEs for image generation; removing them directly degrades quality because the model must learn both high-level structure and low-level detail from raw pixels. RF instead makes representation prediction a native capability: the decoder autoregressively predicts visual representations as intermediate tokens before generating pixels, and these tokens remain in context to guide pixel diffusion within the same backbone. By turning representations from perception outputs into generation targets, RF eliminates dependence on an external generative latent space. Experiments show RF benefits both understanding and generation: pixel-space models with RF match state-of-the-art VAE-based unified models on image generation, and pixel-space RF often outperforms its VAE-based variants on image understanding. Paper: arXiv 2605.31604.

Paper Overview

Field: Computer Vision (CV)
Authors: Yuqing Wang, Zhijie Lin, Ceyuan Yang, Yang Zhao, et al.
Published: 2026-05-29
arXiv: 2605.31604
PDF: 2605.31604.pdf

Background

Unified multimodal models (UMMs) aim to handle both perception and generation tasks within a single model. However, existing UMMs still rely on frozen, separately pretrained VAEs for image generation, which creates a structural bottleneck. Directly removing the VAE leads to quality degradation, because the model must learn both high-level structure and low-level details directly from raw pixels.

Method: Representation Forcing (RF)

Representation Forcing (RF) bridges this gap by making representation prediction a native capability of the model:

RF forces the decoder to autoregressively predict visual representations as intermediate tokens before generating pixels.
These predicted tokens stay in context and guide pixel diffusion within the same backbone.
By converting representations from perception outputs into generation targets, RF removes the dependence on an external generative latent space.

Results

RF benefits both understanding and generation:

Image generation: A pixel-space model trained with RF matches the performance of state-of-the-art VAE-based unified models.
Image understanding: Pixel-space RF generally outperforms its VAE-based variants.