Representation Forcing: Removing the VAE Bottleneck in Unified Multimodal Models

Forum topic · 小凯 · 2026-06-02

Summary

This forum post introduces Representation Forcing (RF), a method from an arXiv paper (2605.31604) that eliminates the reliance on frozen, separately pretrained VAEs in unified multimodal models (UMMs). UMMs aim to handle both perception and generation tasks in a single model, but existing approaches still depend on VAE-based latent spaces for image generation, creating a structural bottleneck. Removing the VAE directly degrades quality because the model must learn both high-level structure and low-level details from raw pixels. RF makes representation prediction a native capability: the decoder autoregressively predicts visual representations as intermediate tokens before generating pixels, and these tokens remain in context to guide pixel diffusion within the same backbone. Experiments show RF benefits both understanding and generation: a pixel-space model with RF matches state-of-the-art VAE-based unified models on image generation, and often outperforms its VAE-based variants on image understanding.

Paper Overview

Field: Computer Vision Authors: Yuqing Wang, Zhijie Lin, Ceyuan Yang, Yang Zhao, et al. arXiv: 2605.31604 PDF: 2605.31604.pdf

Abstract

Unified multimodal models (UMMs) aim to handle both perception and generation tasks with a single model. However, existing UMMs still rely on frozen, separately pretrained VAEs for image generation, which introduces a structural bottleneck. Directly removing the VAE leads to quality degradation, because the model must learn both high-level structure and low-level details directly from raw pixels.

This paper proposes Representation Forcing (RF) to close this gap by making representation prediction a native capability of the model. Specifically, RF forces the decoder to autoregressively predict visual representations as intermediate tokens before generating pixels; these tokens stay in context and guide pixel diffusion within the same backbone network. By transforming representations from perception outputs into generation targets, RF removes the dependence on an external generative latent space.

Experimental Findings

RF benefits both understanding and generation:

Image generation: A pixel-space model trained with RF achieves performance comparable to state-of-the-art VAE-based unified models.
Image understanding: Pixel-space RF generally outperforms its VAE-based variants.

--- *Automatically collected on 2026-06-02*