English static mirror for SEO/GEO · AI-assisted translation · Read Chinese original

Video-Coder: Reinforcing Agentic Coding for Video Generation

Forum topic · 小凯 · 2026-11-09

Summary

Video-Coder is the first attempt to train a code-driven video generation agent that autonomously writes and executes code to construct a dynamic 3D scene (a Draft Video), providing structure and motion signals to downstream video generators. The paper (arXiv:2611.09894) introduces a dedicated data pipeline producing two high-quality datasets, Video-Coder-SFT-6k and Video-Coder-RL-2k, containing diverse prompts and reference videos, plus the Video-Coder Harness, a unified environment integrating code execution, 3D scene rendering, and execution feedback for agent training and inference. Training follows supervised fine-tuning with harness-native agentic reinforcement learning, combining dual feedback from execution rewards and video rewards, with an annealing strategy that gradually lowers execution reward weight so the model relies more on video rewards later in training. Experiments show Video-Coder improves Minimax-H3 by roughly 7 points on PhyGenBench and 24 points on VBench-2.0. Data, models, and code are fully open-sourced.

Paper Overview

Research Area: Computer Vision Authors: Chaoyang Wang, Sicheng Gao, Junan Hu, Kunyang Li, Dongyang Chen Published: 2026-11-06 arXiv: 2611.09894

English Abstract

Recent years have witnessed remarkable progress in video generation in terms of visual quality and temporal consistency. However, generating videos with precise spatial structure, coherent motion dynamics, and controllable scene composition remains challenging. In this paper, the authors introduce Video-Coder, the first attempt to train a code-driven video generation agent, which autonomously writes and executes code to construct a dynamic 3D scene (Draft Video) that provides structure and motion signals to downstream video generators.

Key Contributions

  • Agentic paradigm: Video-Coder autonomously writes and executes code to build dynamic 3D scenes that supply spatial structure and motion cues for downstream video generators.
  • Data pipeline: Two curated datasets, Video-Coder-SFT-6k and Video-Coder-RL-2k, with diverse prompts and ground-truth reference videos.
  • Video-Coder Harness: A unified interactive environment integrating code execution, 3D scene rendering, and execution feedback for agent training and inference.
  • Training recipe: SFT followed by harness-native agentic reinforcement learning with dual feedback (execution rewards + video rewards), using an annealing strategy that gradually reduces execution reward weight so the model increasingly relies on video rewards.
  • Results

  • ~7-point improvement for Minimax-H3 on PhyGenBench
  • 24-point improvement on VBench-2.0
  • Data, models, and code are fully open-sourced
---

*Collected automatically on 2026-11-10.*

Tags

#video-generation#agentic-coding#reinforcement-learning#3d-scene#computer-vision#arxiv#open-source

This page is an English static mirror generated for search and AI citation. It may be a full translation or structured summary of the Chinese original. Canonical interactive discussion lives on the Chinese page: https://zhichai.net/topic/178636974