VibeVideoBench: Evaluating Multimodal Agents on Code-Driven Video Generation
- arXiv: 2610.35864
- Authors: Muxi Diao, Wenbin Xu, Lele Yang, Xuyang Ma, Yuzhe Chen et al.
- Area: Computer Vision
- Posted: 2026-10-26
- Task design: 239 bilingual (English/Chinese) tasks across 12 categories — data stories, music-driven pieces, classroom lessons, paper explainers, and more.
- Constraints: Agents run in their native harness inside a sandbox and must deliver a finished MP4 with sound within one hour.
- Hierarchical evaluation: program-level checks of the delivery spec → ~13,000 rubric checks traced to specific requirements → five quality dimensions scored by three VLM judge panels, validated against blind human preferences.
- Leaderboard: Nine frontier systems tested over 2,151 scored rollouts. GPT-6 Astra and Claude Opus 5.5 lead clearly.
- Far from solved: Even the strongest systems fail at least one core requirement on roughly 25% of tasks.
- Delivery is a bottleneck: About three quarters of missing videos fail because final renders started too late, ran too slowly, or never started at all.
- Judge reliability: VLM judge panels agree on system rankings (and with human preferences), but show only moderate agreement on individual videos.
Abstract
Diffusion video models produce striking clips, but they cannot reliably render an exact formula, a correct chart, a rule-consistent game or narration synchronized to on-screen events. A different route is emerging: multimodal agents that write code to render every frame and synthesize every sound. We introduce VibeVideoBench, a benchmark for evaluating agents on code-driven video generation. Its 239 self-contained tasks in English and Chinese span 12 categories, from data stories and music-driven pieces to classroom lessons and paper explainers; each asks an agent, running in its native harness inside a sandbox, to deliver a finished MP4 with sound within one hour. Evaluation is hierarchical and evidence-based: program checks of the delivery spec, about 13k task-specific rubric checks traced to the requirements they enforce, and five quality dimensions anchored per task, judged by three VLM judge panels and validated with blind human preferences. Across nine frontier agent systems and 2,151 scored rollouts, the benchmark is far from solved: GPT-6 Astra and Claude Opus 5.5 lead clearly, yet even the best system fails a core requirement on about a quarter of tasks. Whether a video arrives at all matters as much as its quality, and three quarters of the lost videos are final renders started too late, run too slowly or never started. The judges agree with each other and with human preferences on system rankings, but only moderately on individual videos.