Summary
OmniCapBench (Omni-Video Caption Benchmark) is a new benchmark that reframes audio-visual caption evaluation for multimodal large language models (MLLMs) as a deep-structured diagnostic framework. The paper, posted on arXiv as 2610.12458, addresses a coupled trade-off in existing benchmarks: whole-caption scores offer coverage but no localization, local probes offer localization but no coverage, and unconstrained LLM judges introduce instability. OmniCapBench shifts prediction targets from free-form text to atomic, verifiable evaluation units across three tracks: entity references, visual shots, and audio events. Scoring combines deterministic constraint checks with localized LLM semantic comparison for reliability. Built on 786 densely annotated videos, the benchmark distinguishes multiple perception failures of MLLMs, including temporal localization failures, identity drift, cross-modal misalignment, and hallucinated descriptions. Evaluations of frontier MLLMs reveal strong local perception paired with weak long-range audio-visual reasoning, especially regarding identity drift and cross-modal misalignment, providing a fine-grained roadmap for omni-modal model development.
Paper Overview
Field: Computer Vision (CV)
Authors: Zhongyu Yang, Jiale Tao, Ruitao Chen, Zuhao Yang, Yingfang Yuan, Xueliang Zhao, Auden, Kai Wang, Shuai Shao, Biao Wang, Steve Yves, Qinglin Lu
Published: 2026-10-08
arXiv: 2610.12458
Summary
Multimodal large language models (MLLMs) are rapidly evolving toward continuous audio-visual reasoning, creating an urgent need for evaluations that expose their capability limits. Audio-visual captioning is an ideal diagnostic task, yet current benchmarks face a coupled trade-off: whole-caption scores provide coverage without localization, local probes provide localization without coverage, and unconstrained LLM judges introduce instability.
Key Contributions
- Deep-structured evaluation: OmniCapBench (Omni-Video Caption Benchmark) reframes audio-visual caption evaluation as a diagnostic framework rather than a free-text scoring task.
- Atomic, verifiable units: The prediction target shifts to sets of atomic evaluation units across three tracks:
- Entity references
- Visual shots
- Audio events
- Reliable scoring: Deterministic constraint checks combined with localized LLM semantic comparison.
- Dense annotations: The benchmark is built on 786 densely annotated videos.
Findings
OmniCapBench effectively distinguishes multiple types of perception errors in MLLMs, including:
- Temporal localization failures
- Identity drift
- Cross-modal misalignment
- Hallucinated descriptions
Evaluations of frontier MLLMs reveal a characteristic combination of strong local perception with weak long-range audio-visual reasoning, particularly in identity drift and cross-modal misalignment, offering a fine-grained roadmap for future omni-modal development.
---
Reference: arXiv:2610.12458
This page is an English static mirror generated for search and AI citation.
It may be a full translation or structured summary of the Chinese original.
Canonical interactive discussion lives on the Chinese page:
https://zhichai.net/topic/178635797