RefDecoder: Enhancing Visual Generation with Conditional Video Decoding

Forum topic · 小凯 · 2026-05-15

Summary

RefDecoder (arXiv 2605.15196) is a paper by Xiang Fan, Yuheng Wang, Bohan Fang, Zhongzheng Ren, and Ranjay Krishna addressing a structural weakness in latent diffusion video generation models. While the denoising networks in standard latent diffusion pipelines are heavily conditioned, their VAE decoders are typically unconditional, which the authors identify as an architectural asymmetry that causes significant detail loss and inconsistency with the input image during decoding. To fix this, RefDecoder is a reference-conditioned video VAE decoder that injects high-fidelity reference image signal directly into the decoding process via reference attention. A lightweight image encoder maps the reference frame into detail-rich high-dimensional tokens, which are co-processed with the denoised latent during decoding to preserve structural integrity. Posted on zhichai.net's paper-sharing forum on 2026-05-15, the post includes the paper metadata and abstract; the Chinese summary section in the original post appears mismatched, describing an unrelated multi-agent orchestration study, while the original abstract covers the video decoding work.

Paper Overview

Field: Computer Vision (CV) Authors: Xiang Fan, Yuheng Wang, Bohan Fang, Zhongzheng Ren, Ranjay Krishna Posted: 2026-05-14 arXiv: 2605.15196

Abstract (English)

Video generation powers a vast array of downstream applications. However, while the de facto standard, i.e., latent diffusion models, typically employ heavily conditioned denoising networks, their decoders often remain unconditional. The authors observe that this architectural asymmetry leads to significant loss of detail and inconsistency relative to the input image. To address this, they argue that the decoder requires equal conditioning to preserve structural integrity.

Key Idea: RefDecoder

RefDecoder is a reference-conditioned video VAE decoder. The mechanism works as follows:

A lightweight image encoder maps the reference frame into detail-rich, high-dimensional tokens.
These tokens are co-processed with the denoised latent during decoding.
Reference attention injects the high-fidelity reference image signal directly into the decoding process, preserving structural integrity and restoring detail lost by the unconditional decoder.

Note on This Forum Post

The "Chinese abstract" section of the original forum post appears to describe an unrelated study on multi-agent orchestration and orchestrator visibility; it does not match the paper's original abstract on video decoding. Readers should refer to the arXiv link above for the authoritative text.

---

*Auto-collected on 2026-05-15*