Paper Overview
Research area: Computer Vision Authors: Binxu Li, Haoyi Duan, Yuhui Zhang, Yaohui Zhang, Zihao Lin, Kaituo Feng, Suozhi Huang, Xiangyi Li, Yu Li, Chunyuan Li, Shilong Liu, Mengdi Wang arXiv: 2609.24997
Summary
Recent advances in video generative models have enabled high-fidelity, temporally coherent video generation. However, these models often struggle to satisfy prompts requiring specialized knowledge, specific identities, physical consistency, or ordered events. VideoGen-Agent addresses this by training a multimodal agent, via multitask agentic reinforcement learning, to use external tools during video generation.
The agent coordinates augmentation, generation, and verification tools through multi-turn interactions, using the prompt and intermediate observations to guide its decisions.
Training Approach
- A shared policy is trained on a category-balanced dataset spanning six tasks.
- Supervised fine-tuning on teacher-generated trajectories establishes tool-use behavior.
- The policy is then refined through reinforcement learning.
- A category-aware hybrid reward simultaneously evaluates tool-call effectiveness, task-appropriate tool selection, and the quality of generated videos.
- Procedural knowledge
- Single-entity and multi-entity identity preservation
- Physical consistency
- Scene composition
- Multi-shot temporal structure
- On VABench, VideoGen-Agent improves over its base text-to-video generator by 19.1 points (from 56.5 to 75.6).
- Without additional agent training, simply upgrading the generation tool further increases the score to 86.1.
- Human evaluators prefer the upgraded configuration over the strongest standalone baseline in 84.3% of comparisons.
VABench Benchmark
The authors propose VABench, a held-out benchmark of 600 prompts covering:
Results
--- *Collected automatically on 2026-09-23.*