OmniVideo-100K: A Dataset for Audio-Visual Reasoning via Entity-Anchored Video Scripting

Forum topic · 小凯 · 2026-06-16

Summary

OmniVideo-100K is a new instruction-tuning dataset for audio-visual question answering, introduced in arXiv paper 2606.14702. Existing automated pipelines follow a video-caption-QA paradigm that segments videos into short clips and separately describes audio and visual modalities, severing sound-source associations and producing inconsistent entity descriptions across segments. The authors propose an automated data engine with two mechanisms: (1) Entity-Anchored Video Scripting, which converts videos into structured scripts containing summaries, entity lists, and segmented audio-visual descriptions, using entity lists as global priors for cross-segment consistency; and (2) Cue-Guided QA Generation, which mines cross-segment multimodal cues before generating QA pairs. Using this pipeline, the team built the OmniVideo-100K training set and a human-verified test set, OmniVideo-Test. Fine-tuning VITA-1.5, Qwen2.5-Omni-7B, and Qwen3-Omni-30B yields up to 20.59% improvement on OmniVideo-Test and up to 12.64% gains on benchmarks like Daily-Omni and JointAVBench.

OmniVideo-100K: A Dataset for Audio-Visual Reasoning through Structured Video Scripting

Field: Computer Vision (CV) Authors: Xinyue Cai, Chaoyou Fu, Yi-Fan Zhang arXiv: 2606.14702

Abstract

Current automated pipelines for audio-visual Question Answering (QA) generally adopt a "video-caption-QA" paradigm. However, these methods typically segment videos into short clips and generate separate descriptions for audio and visual modalities. This decoupled processing severs inherent associations between sounds and their visual sources, while independent clip processing often causes inconsistent descriptions of the same entity across segments. Furthermore, coupling long-text comprehension and QA synthesis into a single step often restricts models to localized events, yielding questions lacking long-term temporal connections and deep cross-modal reasoning.

Proposed Approach

The authors propose an automated data engine featuring two mechanisms:

1. Entity-Anchored Video Scripting — converts videos into structured scripts comprising a summary, a list of main entities, and segmented audio-visual descriptions. The entity list serves as a global prior, ensuring consistent cross-segment references and restoring audio-visual associations. 2. Cue-Guided QA Generation — prompts the model to first mine cross-segment, multimodal cues from the script, then generate QA pairs grounded in these high-value cues.

Results

Using this pipeline, the authors construct:

OmniVideo-100K: a large-scale instruction fine-tuning dataset
OmniVideo-Test: a human-verified test set

Fine-tuning VITA-1.5, Qwen2.5-Omni-7B, and Qwen3-Omni-30B on OmniVideo-100K achieves up to 20.59% improvement on OmniVideo-Test, with strong generalization on benchmarks such as Daily-Omni and JointAVBench (up to 12.64% improvement).

--- *Source: zhichai.net forum post, auto-collected 2026-06-16.*