Paper Overview
- Field: Computer Vision (CV)
- Authors: Yongsheng Yu, Ziyun Zeng, Zhiyuan Xiao
- Published: 2026-05-19
- arXiv: 2505.14307
- Agentic editing framework: Decouples *intent resolution* (handled by the VLM agent) from *video generation* (handled by the diffusion transformer), allowing the editing model to receive fully specified conditioning inputs.
- Training recipe: Supervised training for full edit planning and reference image selection, combined with preference-pair optimization for tool-use robustness and instruction refinement.
- AgentEdit-Bench: A new benchmark specifically targeting underspecified editing requests that lack ready-made prompts, references, or spatial localizations.
- Transferability: The trained VLM agent can be transferred to compatible frozen video editing models without retraining the underlying diffusion transformer.
- Paper: https://arxiv.org/abs/2505.14307
- Project page: https://yeates.github.io/Aurora-Page
Summary
Recent video editing models have converged on a unified conditioning design: a single diffusion transformer jointly consumes text, source video, and reference images, and one set of weights covers replacement, removal, style transfer, and reference-driven insertion. The design is flexible, but it assumes that the user already provides model-ready text, reference images, and spatial grounding for local edits, which real requests often omit.
Aurora is an agentic video editing framework that pairs a tool-augmented vision-language model (VLM) agent with a unified video diffusion transformer. The VLM agent maps a raw user request to a structured edit plan aligned with the transformer's conditioning channels, thereby resolving textual and visual underspecification before generation.
The VLM agent is trained with supervised data to produce complete edit plans and select reference images, while preference pairs are used to enhance the robustness of tool use and instruction refinement.
The authors introduce AgentEdit-Bench, a benchmark designed to evaluate agent-enhanced video editing under text and visual underspecification.
Key Contributions
Experimental Results
Experiments on AgentEdit-Bench and two existing video editing benchmarks show that Aurora outperforms instruction-only baselines. The transferability of the VLM agent to compatible frozen video editing models is also demonstrated, suggesting that the agentic layer is a reusable, model-agnostic front-end for unified video editing systems.