RefAlign: Representation Alignment for Reference-to-Video Generation

Forum topic · 小凯 · 2026-03-28

Summary

RefAlign is a representation alignment framework for reference-to-video (R2V) generation, a controllable video synthesis paradigm that combines text prompts with reference images for applications like personalized advertising and virtual try-on. Existing R2V methods feed auxiliary high-level semantic or cross-modal features, alongside the VAE latent of the reference image, into a diffusion Transformer (DiT). While these help with pixel-level information leakage in the VAE latent space, they still suffer from copy-paste artifacts and multi-subject confusion caused by modality mismatch across heterogeneous encoders. RefAlign explicitly aligns the DiT reference-branch features to the semantic space of a vision foundation model (VFM). Its core is a reference alignment loss that pulls together reference and VFM features of the same subject to improve identity consistency, while pushing apart features of different subjects to enhance semantic distinguishability. The strategy is applied only during training, adding no inference-time overhead, and achieves a better balance between text controllability and reference fidelity. Experiments on the OpenS2V-Eval benchmark show RefAlign outperforms state-of-the-art methods in TotalScore. Paper: arXiv 2603.25743.

Paper Overview

Field: Computer Vision Authors: Lei Wang, YuXin Song, Ge Wu, Haocheng Feng, Hang Zhou, Jingdong Wang, Yaxing Wang, Jian Yang Published: 2026-03-26 arXiv: 2603.25743

Abstract

Reference-to-video (R2V) generation is a controllable video synthesis paradigm that constrains the generation process using both text prompts and reference images, enabling applications such as personalized advertising and virtual try-on. In practice, existing R2V methods typically introduce additional high-level semantic or cross-modal features alongside the VAE latent representation of the reference image and jointly feed them into the diffusion Transformer (DiT). These auxiliary representations provide semantic guidance and act as implicit alignment signals, which can partially alleviate pixel-level information leakage in the VAE latent space. However, they may still struggle to address copy-paste artifacts and multi-subject confusion caused by modality mismatch across heterogeneous encoders.

Method

This paper proposes RefAlign, a representation alignment framework that explicitly aligns the DiT reference-branch features to the semantic space of a vision foundation model (VFM). At its core is a reference alignment loss that:

Pulls together the reference features and VFM features of the same subject to improve identity consistency
Pushes apart corresponding features of different subjects to enhance semantic distinguishability

This simple yet effective strategy is applied only during training, introduces no inference-time overhead, and achieves a better balance between text controllability and reference fidelity.

Results

Extensive experiments on the OpenS2V-Eval benchmark show that RefAlign outperforms current state-of-the-art methods in TotalScore, validating the effectiveness of explicit reference alignment for the R2V task.

---

*Auto-collected on 2026-03-28*