[论文] OmniNFT: Modality-wise Omni Diffusion Reinforcement for Joint Audio-Vi...

## 论文概要 **研究领域**: CV **作者**: Guohui Zhang, XiaoXiao Ma, Jie Huang, Hang Xu, Hu Yu, Siming Fu, Yuming Li, Zeyue Xue, Lin Song, Haoyang Huang, Nan Duan, Feng Zhao **发布时间**: 2026-05-12 **arXiv**: [2605.12480](https://arxiv.org/abs/2605.12480) ## 中文摘要 联合音视频生成近期取得显著进展，但真实应用要求强模态内保真度、跨模态对齐和细粒度同步。强化学习（RL）提供有前景的范式，但将其扩展到多目标多模态联合音视频生成仍未被探索。深入分析首先揭示应用 RL 的主要障碍：（i）多目标优势不一致，多模态输出的优势在组内不总一致；（ii）多模态梯度不平衡，视频分支梯度泄漏到负责模态内生成的浅音频层；（iii）统一信用分配，细粒度跨模态对齐区域未能有效探索。这些缺陷表明，具有单一全局优势的朴素 RL 微调策略常导致次优结果。为此，我们提出 OmniNFT，模态感知在线扩散 RL 框架，三项关键创新：（1）模态级优势路由，将独立每奖励优势路由到各自模态生成分支。（2）层间梯度手术，选择性分离浅音频层上的视频分支梯度同时保留跨模态交互层梯度。（3）区域级损失重加权，将策略优化调节向与音视频同步和细粒度对齐相关的关键区域。JavisBench 和 VBench 上以 LTX-2 骨干的实验表明，OmniNFT 在音频和视频感知质量、跨模态对齐和音视频同步方面实现全面提升。 ## 原文摘要 Recent advances in joint audio-video generation have been remarkable, yet real-world applications demand strong per-modality fidelity, cross-modal alignment, and fine-grained synchronization. Reinforcement Learning (RL) offers a promising paradigm, but its extension to multi-objective and multi-modal joint audio-video generation remains unexplored. Notably, our in-depth analysis first reveals that the primary obstacles to applying RL in this stem from: (i) multi-objective advantages inconsistency, where the advantages of multimodal outputs are not always consistent within a group; (ii) multi-modal gradients imbalance, where video-branch gradients leak into shallow audio layers responsible for intra-modal generation; (iii) uniform credit assignment, where fine-grained cross-modal alignment ... --- *自动采集于 2026-05-14* #论文 #arXiv #CV #小凯