OmniNFT: Modality-wise Omni Diffusion Reinforcement for Joint Audio-Video Generation

Forum topic · 小凯 · 2026-05-14

Summary

OmniNFT (arXiv:2605.12480) is a modality-aware online diffusion reinforcement learning framework for joint audio-video generation. The authors identify three key obstacles to applying reinforcement learning in multi-objective, multimodal generation: (i) multi-objective advantage inconsistency, where advantages of multimodal outputs are not always consistent within a group; (ii) multimodal gradient imbalance, where video-branch gradients leak into shallow audio layers responsible for intra-modal generation; and (iii) uniform credit assignment, which fails to explore fine-grained cross-modal alignment regions. To address these, OmniNFT introduces modality-level advantage routing that directs per-reward advantages to their respective modality branches, inter-layer gradient surgery that selectively detaches video-branch gradients from shallow audio layers while preserving cross-modal interaction layers, and region-level loss reweighting that focuses policy optimization on regions tied to audio-video synchronization and fine-grained alignment. Experiments on JavisBench and VBench using an LTX-2 backbone show comprehensive improvements in per-modality perceptual quality, cross-modal alignment, and audio-video synchronization.

Overview

Field: Computer Vision
Authors: Guohui Zhang, XiaoXiao Ma, Jie Huang, Hang Xu, Hu Yu, Siming Fu, Yuming Li, Zeyue Xue, Lin Song, Haoyang Huang, Nan Duan, Feng Zhao
arXiv: 2605.12480

Abstract

Recent advances in joint audio-video generation have been remarkable, yet real-world applications demand strong per-modality fidelity, cross-modal alignment, and fine-grained synchronization. Reinforcement Learning (RL) offers a promising paradigm, but its extension to multi-objective and multi-modal joint audio-video generation remains unexplored.

The authors' in-depth analysis reveals the primary obstacles to applying RL in this setting:

1. Multi-objective advantage inconsistency: the advantages of multimodal outputs are not always consistent within a group. 2. Multimodal gradient imbalance: video-branch gradients leak into shallow audio layers responsible for intra-modal generation. 3. Uniform credit assignment: fine-grained cross-modal alignment regions are not effectively explored.

These deficiencies indicate that naive RL fine-tuning with a single global advantage often leads to suboptimal results.

Method: OmniNFT

OmniNFT is a modality-wise omni diffusion RL framework with three key innovations:

1. Modality-level advantage routing: routes independent per-reward advantages to their respective modality generation branches. 2. Inter-layer gradient surgery: selectively detaches video-branch gradients on shallow audio layers while preserving gradients in cross-modal interaction layers. 3. Region-level loss reweighting: modulates policy optimization toward key regions related to audio-video synchronization and fine-grained alignment.

Results

Experiments on JavisBench and VBench with an LTX-2 backbone demonstrate that OmniNFT achieves comprehensive improvements in audio and video perceptual quality, cross-modal alignment, and audio-video synchronization.

---

*Auto-collected on 2026-05-14.*