[论文] OmniCapBench: A Deep-Structured Evaluation Framework for Fine-Grained ...

论文概要

研究领域: CV 作者: Zhongyu Yang, Jiale Tao, Ruitao Chen, Zuhao Yang, Yingfang Yuan, Xueliang Zhao, Auden, Kai Wang, Shuai Shao, Biao Wang, Steve Yves, Qinglin Lu 发布时间: 2026-10-08 arXiv: 2610.12458

中文摘要

多模态大语言模型（MLLM）正快速向连续音视频推理演进，迫切需要能暴露其能力边界的评估手段。音视频描述是理想的诊断任务，但现有基准面临耦合的权衡：整段描述评分提供覆盖度却缺乏定位，局部探测提供定位却缺乏覆盖度，而无约束的 LLM 评判又引入不稳定性。我们提出 OmniCapBench（Omni-Video Caption Benchmark），将音视频描述评估重构为深层结构化诊断框架。OmniCapBench 将预测目标从自由文本转为跨三条赛道的原子化可验证评估单元：实体指称、视觉镜头和音频事件，通过确定性约束检查和局部 LLM 语义比较实现可靠评分。凭借 786 个稠密标注视频，OmniCapBench 有效区分 MLLM 的多种感知错误，包括时间定位失败、身份漂移、跨模态错位和幻觉描述。对前沿 MLLM 的评估揭示了强局部感知与弱长程音视频推理并存的特点，特别是在身份漂移和跨模态错位方面，为全模态发展提供了细粒度路线图。

原文摘要

Multimodal large language models (MLLMs) are rapidly evolving toward continuous audio--visual reasoning, creating an urgent need for evaluations that expose their capability limits. Audio--visual captioning is an ideal diagnostic task, yet current benchmarks face a coupled trade-off: whole-caption scores provide coverage without localization, local probes provide localization without coverage, and unconstrained LLM judges introduce instability. We introduce OmniCapBench (Omni-Video Caption Benchmark), a benchmark that reframes audio--visual caption evaluation as a deep-structured diagnostic framework. OmniCapBench shifts the prediction target from free-form text to sets of atomic, verifiable evaluation units across three tracks: entity references, visual shots, and audio events, enabling r...

#论文 #arXiv #CV #小凯