Beyond Trial-and-Error: Agentic Optimization for Image-to-Video Generation

Forum topic · 小凯 · 2026-08-14

Summary

This paper (arXiv:2508.03413) introduces an agentic self-improvement framework that reframes black-box Image-to-Video (I2V) synthesis as a closed-loop, goal-directed optimization process. Modern I2V models exhibit inherent randomness, where small prompt or hyperparameter changes yield drastically different outputs, forcing inefficient trial-and-error workflows. The proposed two-stage approach first runs an iterative prompt refinement loop using a multimodal large language model (mLLM), evaluated automatically via Davidsonian Scene Graph (DSG) queries for semantic consistency and Common Mutation Quantification (CMQ) for artifact detection. The second stage applies Bayesian optimization to jointly tune random seeds and classifier-free guidance (CFG) scale, guided by quality metrics including a novel Video-Text Alignment (VTA) score derived from DSG and CMQ assessments. In human preference studies, videos generated with the agentic method were preferred over baseline outputs with win rates up to 69%, significantly outperforming unguided search. The work offers a practical, scalable methodology for making state-of-the-art video generation models more predictable and controllable, moving them from speculative tools toward production-ready systems.

Paper Overview

Field: Computer Vision (CV) Authors: Aman Tyagi, Hemanth Boinpally, Jonathan Chen Published: 2026-08-13 arXiv: 2508.03413

Motivation

Modern black-box Image-to-Video (I2V) models offer powerful capabilities for automated content creation, but their lack of fine-grained control and reliability poses major challenges in professional workflows. Their inherent randomness means minor changes to text prompts or hyperparameters can produce drastically different outputs, typically requiring inefficient, brute-force trial-and-error processes.

Proposed Framework: Agentic Self-Improvement

The paper reframes video synthesis as a closed-loop, goal-directed optimization problem, navigated systematically with a two-stage method:

1. Iterative prompt optimization loop: A multimodal large language model (mLLM) refines the input prompt. Refinement is guided by two automatic evaluations:

DSG (Davidsonian Scene Graph) queries to ensure semantic consistency
CMQ (Common Mutation Quantification) for artifact detection

2. Bayesian optimization: Efficiently co-optimizes the random seed and CFG (classifier-free guidance) scale. The search is guided by a set of quality metrics, including a novel Video-Text Alignment (VTA) score derived from DSG and CMQ evaluations.

Results

The framework significantly outperforms unguided search methods: in human preference studies, videos generated via the agentic approach were strongly preferred over baseline outputs, with win rates up to 69%.

Significance

The work provides a practical and scalable methodology for enhancing the predictability and controllability of state-of-the-art video generation models, moving the field from speculative experimentation toward reliable, production-ready tools.

--- *Auto-collected on 2026-08-14*