RefAlign: Representation Alignment for Reference-to-Video Generation

Forum topic · 小凯 · 2026-03-28

Summary

RefAlign is a representation alignment framework for reference-to-video (R2V) generation, a controllable video synthesis paradigm that uses text prompts and reference images for applications like personalized advertising and virtual try-on. Existing R2V methods feed auxiliary high-level semantic or cross-modal features alongside the reference image's VAE latents into a diffusion Transformer (DiT), but still suffer from copy-paste artifacts and multi-subject confusion caused by modality mismatch across heterogeneous encoders. RefAlign explicitly aligns DiT reference-branch features to the semantic space of a vision foundation model (VFM). Its core reference alignment loss pulls together reference and VFM features of the same subject to improve identity consistency while pushing apart features of different subjects to enhance semantic distinguishability. The loss is applied only during training, adding no inference-time overhead, and yields a better balance between text controllability and reference fidelity. Experiments on the OpenS2V-Eval benchmark show RefAlign outperforms current state-of-the-art methods in TotalScore.

Paper Overview

Field: Computer Vision Authors: Lei Wang, YuXin Song, Ge Wu, Haocheng Feng, Hang Zhou, Jingdong Wang, Yaxing Wang, Jian Yang arXiv: 2603.25743

Background

Reference-to-video (R2V) generation is a controllable video synthesis paradigm that constrains the generation process using both text prompts and reference images, enabling applications such as personalized advertising and virtual try-on.

In practice, existing R2V methods typically introduce additional high-level semantic or cross-modal features alongside the VAE latent representation of the reference image, jointly feeding them into the diffusion Transformer (DiT). These auxiliary representations provide semantic guidance and act as implicit alignment signals, which can partially alleviate pixel-level information leakage in the VAE latent space. However, they may still struggle to address copy-paste artifacts and multi-subject confusion caused by modality mismatch across heterogeneous encoders.

Proposed Method

The paper proposes RefAlign, a representation alignment framework that explicitly aligns DiT reference-branch features to the semantic space of a vision foundation model (VFM).

At its core is a reference alignment loss that:

Pulls together reference features and VFM features of the same subject, improving identity consistency.
Pushes apart corresponding features of different subjects, enhancing semantic distinguishability.

Key Advantages

Simple yet effective strategy, applied only during training — no inference-time overhead.
Achieves a better balance between text controllability and reference fidelity.

Results

Extensive experiments on the OpenS2V-Eval benchmark show that RefAlign outperforms current state-of-the-art methods in TotalScore, validating the effectiveness of explicit reference alignment for R2V tasks.