Summary
This paper introduces an agentic self-improving framework that reframes black-box Image-to-Video (I2V) generation as a closed-loop, goal-directed optimization problem, addressing the unreliability and lack of fine-grained control of current models. The two-stage approach first uses a multimodal LLM to iteratively refine input prompts, validated by Davidsonian Scene Graph (DSG) queries for semantic consistency and Commonsense Misaligned Queries (CMQ) for artifact detection. In the second stage, Bayesian optimization jointly tunes the random seed and CFG scale, guided by quality metrics including a novel Video-Text Alignment (VTA) score derived from DSG and CMQ evaluations. In human preference studies, videos produced by the agentic method were strongly preferred over baseline outputs, with win rates up to 69%. The work provides a practical, scalable methodology for moving I2V from speculative experimentation toward production-ready creative tools.
Paper Overview
- Field: Computer Vision (CV)
- Authors: Aman Tyagi, Hemanth Boinpally, Jonathan Chen
- Published: 2026-08-13
- arXiv: 2508.03413
Summary (English)
Modern black-box Image-to-Video (I2V) models offer powerful capabilities for automated content creation, but their lack of fine-grained control and reliability poses significant challenges in professional workflows. Their inherent stochasticity causes minor changes in text prompts or hyperparameters to produce drastically different outputs, typically requiring inefficient, brute-force trial-and-error.
To address these limitations, the paper introduces an agentic self-improving framework that reframes video synthesis as a closed-loop, goal-directed optimization. The framework uses a novel two-stage method to systematically navigate the generation parameter space.
Stage 1: Iterative Prompt Refinement
A multimodal Large Language Model (mLLM) iteratively refines input prompts. This refinement enables two automatic evaluations:
- Davidsonian Scene Graph (DSG) queries to ensure semantic consistency.
- Commonsense Misaligned Queries (CMQ) for artifact detection.
Stage 2: Bayesian Optimization of Generation Parameters
Bayesian optimization efficiently co-optimizes the random seed and classifier-free guidance (CFG) scale. The search is guided by a set of quality metrics, including a novel Video-Text Alignment (VTA) score derived from the DSG and CMQ evaluations.Results
The framework significantly outperforms unguided search methods. In human preference studies, videos generated with the agentic method were strongly preferred over baseline outputs, achieving win rates of up to 69%. This work provides a practical and scalable methodology for enhancing the predictability and control of state-of-the-art video generation models, moving the field from speculative curiosity toward reliable, production-ready tools.Key Points
- Problem: Black-box I2V models are stochastic and offer little fine-grained control, forcing practitioners into inefficient trial-and-error.
- Solution: A two-stage agentic optimization framework that automates prompt refinement and hyperparameter tuning.
- Stage 1 Tools: mLLM-driven prompt refinement, validated via DSG (semantic consistency) and CMQ (artifact detection).
- Stage 2 Tools: Bayesian optimization over seed and CFG scale, guided by a new VTA score.
- Outcome: Up to 69% win rate vs. baseline in human preference studies, with substantially improved controllability and reproducibility.
Source
- arXiv: https://arxiv.org/abs/2508.03413
This page is an English static mirror generated for search and AI citation.
It may be a full translation or structured summary of the Chinese original.
Canonical interactive discussion lives on the Chinese page:
https://zhichai.net/topic/178633460