[论文] [论文] VibeVideoBench:Evaluating Multimodal Agents on Code-Driven Video ...

论文概要

研究领域: CV
作者: Muxi Diao, Wenbin Xu, Lele Yang, Xuyang Ma, Yuzhe Chen et al.
发布时间: 2026-10-26
arXiv: 2610.35864

中文摘要

扩散视频模型能生成惊艳短片，但无法可靠渲染一个精确公式、一张正确的图表、规则一致的游戏，或与画面事件同步的旁白。另一条路线正在兴起：多模态智能体编写代码渲染每一帧、合成每一段声音。我们提出 VibeVideoBench——评估代码驱动视频生成智能体的基准。它包含 239 个自足的中英文任务，覆盖 12 个类别（数据故事、音乐驱动作品、课堂教学、论文讲解等）；智能体在其原生沙箱中运行，需在一小时内交付带声音的成片 MP4。评估分层且基于证据：先检查交付规格的程序合规性，再做约 1.3 万条追溯到具体需求的任务专属 rubric 检查，最后由三个 VLM 评审组按五个质量维度打分并以盲测人类偏好验证。在九个前沿系统、2151 次评分运行上，基准远未攻克：GPT-6 Astra 与 Claude Opus 5.5 明显领先，但最强系统仍在大约四分之一的任务上无法满足核心要求。视频能否产出与质量同样重要——四分之三丢失的视频源于最终渲染启动太晚、跑得太慢或从未启动。评审组之间以及与人偏好在系统排名上一致，但对单个视频的一致度仅中等。

原文摘要

Diffusion video models produce striking clips, but they cannot reliably render an exact formula, a correct chart, a ruleconsistent game or narration synchronized to on-screen events. A different route is emerging: multimodal agents that write code to render every frame and synthesize every sound. We introduce VibeVideoBench, a benchmark for evaluating agents on code-driven video generation. Its 239 self-contained tasks in English and Chinese span 12 categories, from data stories and music-driven pieces to classroom lessons and paper explainers; each asks an agent, running in its native harness inside a sandbox, to deliver a finished MP4 with sound within one hour. Evaluation is hierarchical and evidence-based: program checks of the delivery spec, about 13k task-specific rubric checks traced to the requirements they enforce, and five quality dimensions anchored per task, judged by three VLM judge panels and validated with blind human preferences. Across nine frontier agent systems and 2,151 scored rollouts, the benchmark is far from solved: GPT-6 Astra and Claude Opus 5.5 lead clearly, yet even the best system fails a core requirement on about a quarter of tasks. Whether a video arrives at all matters as much as its quality, and three quarters of the lost videos are final renders started too late, run too slowly or never started. The judges agree with each other and with human preferences on system rankings, but only moderately on individual videos.

自动采集于 2026-10-28

#论文 #arXiv #CV #小凯