Aurora: Unified Video Editing with a Tool-Using Agent
- Field: Computer Vision
- Authors: Yongsheng Yu, Ziyun Zeng, Zhiyuan Xiao
- arXiv: 2505.14307
- Project page: https://yeates.github.io/Aurora-Page
- Paper: https://arxiv.org/abs/2505.14307
- Project page: https://yeates.github.io/Aurora-Page
Motivation
Recent video editing models have converged on a unified conditioning design: a single diffusion transformer jointly consumes text, source video, and reference images, and one set of weights covers replacement, removal, style transfer, and reference-driven insertion. The design is flexible, but it assumes that the user already provides model-ready text, reference images, and spatial grounding for local edits — information that real requests often omit.
Approach
Aurora pairs a tool-augmented vision-language model (VLM) agent with a unified video diffusion transformer. The VLM agent maps a raw user request to a structured edit plan aligned with the transformer's conditioning channels, thereby resolving textual and visual underspecification before generation. The agent is trained with supervised data for full edit planning and reference image selection, and with preference pairs to enhance tool-use robustness and instruction refinement.
Evaluation
The authors introduce AgentEdit-Bench, a benchmark for evaluating agentic video editing under textual and visual underspecification. Experiments on AgentEdit-Bench and two existing video editing benchmarks show that Aurora outperforms instruction-only baselines. Additionally, the VLM agent transfers to compatible frozen video editing models.