Paper Overview
Research Area: Computer Vision Authors: Chaoyang Wang, Sicheng Gao, Junan Hu, Kunyang Li, Dongyang Chen Published: 2026-11-06 arXiv: 2611.09894
English Abstract
Recent years have witnessed remarkable progress in video generation in terms of visual quality and temporal consistency. However, generating videos with precise spatial structure, coherent motion dynamics, and controllable scene composition remains challenging. In this paper, the authors introduce Video-Coder, the first attempt to train a code-driven video generation agent, which autonomously writes and executes code to construct a dynamic 3D scene (Draft Video) that provides structure and motion signals to downstream video generators.
Key Contributions
- Agentic paradigm: Video-Coder autonomously writes and executes code to build dynamic 3D scenes that supply spatial structure and motion cues for downstream video generators.
- Data pipeline: Two curated datasets, Video-Coder-SFT-6k and Video-Coder-RL-2k, with diverse prompts and ground-truth reference videos.
- Video-Coder Harness: A unified interactive environment integrating code execution, 3D scene rendering, and execution feedback for agent training and inference.
- Training recipe: SFT followed by harness-native agentic reinforcement learning with dual feedback (execution rewards + video rewards), using an annealing strategy that gradually reduces execution reward weight so the model increasingly relies on video rewards.
- ~7-point improvement for Minimax-H3 on PhyGenBench
- 24-point improvement on VBench-2.0
- Data, models, and code are fully open-sourced
Results
*Collected automatically on 2026-11-10.*