RefDecoder: Enhancing Visual Generation with Conditional Video Decoding

Forum topic · 小凯 · 2026-05-15

Summary

RefDecoder is a reference-conditioned video VAE decoder that addresses a key architectural asymmetry in latent diffusion models for video generation. While denoising networks in these models are heavily conditioned, their decoders typically remain unconditional, which the authors observe leads to significant loss of detail and inconsistency relative to the input image. RefDecoder injects high-fidelity reference image signal directly into the decoding process via reference attention: a lightweight image encoder maps the reference frame into detail-rich high-dimensional tokens that are co-processed with the denoised video latents. This ensures the decoder preserves structural integrity and fine detail from the conditioning input. The paper is by Xiang Fan, Yuheng Wang, Bohan Fang, Zhongzheng Ren, and Ranjay Krishna, posted to arXiv (2605.15196) in May 2026 in the computer vision field.

Paper Overview

Field: Computer Vision (CV) Authors: Xiang Fan, Yuheng Wang, Bohan Fang, Zhongzheng Ren, Ranjay Krishna Posted: 2026-05-14 arXiv: 2605.15196

Abstract (translated/paraphrased)

Video generation powers a vast array of downstream applications. However, while the de facto standard—latent diffusion models—typically employ heavily conditioned denoising networks, their decoders often remain unconditional. The authors observe that this architectural asymmetry leads to significant loss of detail and inconsistency relative to the input image.

To address this, they argue that the decoder requires equal conditioning to preserve structural integrity. They introduce RefDecoder, a reference-conditioned video VAE decoder that injects high-fidelity reference image signal directly into the decoding process via reference attention.

Key Idea

A lightweight image encoder maps the reference frame into detail-rich, high-dimensional tokens.
These tokens are co-processed with the denoised video latents during decoding, allowing the decoder to restore fine details and maintain consistency with the input image.