Paper Overview
Field: Computer Vision (CV) Authors: Xinyue Cai, Chaoyou Fu, Yi-Fan Zhang Published: 2026-06-12 arXiv: 2606.14702
Summary
Current automated pipelines for audio-visual Question Answering (QA) generally adopt a "video-caption-QA" paradigm. However, these methods typically segment videos into short clips and generate separate descriptions for audio and visual modalities. This decoupled processing severs inherent associations between sounds and their visual sources, while independent clip processing often causes inconsistent descriptions of the same entity across segments. Furthermore, coupling long-text comprehension and QA synthesis into a single step often restricts models to localized events, yielding questions lacking long-term temporal connections and deep cross-modal reasoning.
Proposed Data Engine
To address these issues, the authors propose an automated data engine with two mechanisms:
1. Entity-Anchored Video Scripting — converts videos into structured scripts including a summary, a list of main entities, and segmented audio-visual descriptions. The entity list serves as a global prior, ensuring consistent cross-segment references and rebuilding audio-visual associations. 2. Clue-Guided QA Generation — prompts the model to first mine cross-segment, multi-modal clues from the script, then generate QA pairs based on these high-value clues.
Results
Using this pipeline, the authors constructed the instruction-tuning dataset OmniVideo-100K and a human-verified test set OmniVideo-Test. Fine-tuning VITA-1.5, Qwen2.5-Omni-7B, and Qwen3-Omni-30B on OmniVideo-100K achieves up to 20.59% improvement on OmniVideo-Test, along with strong generalization gains (up to 12.64%) on benchmarks such as Daily-Omni and JointAVBench.
---
*Auto-collected on 2026-06-16*