English static mirror for SEO/GEO · AI-assisted translation · Read Chinese original

AVA-Encoder: Towards Agent-Native Video Representation Learning

Forum topic · 小凯 · 2026-08-14

Summary

AVA-Encoder (arXiv 2508.03420) is a framework from Chuyue Li, Jinpeng Yu, and Haozhe Wang that learns agent-native video representations to help creative agents produce cinematic videos from high-quality human films. The method converts videos into knowledge graph (KG) representations and then reconstructs them: hierarchical and state nodes store structured text, an asset layer holds generated images, audio, and video, and typed edges preserve relations between descriptions and assets in a form agents can easily query and edit. Video reconstruction error drives a text-gradient optimization framework that expresses evaluation feedback as natural-language update directions for pseudo-training of a data-independent encoding policy (outer loop) and test-time refinement of the data-dependent KG representation (inner loop). Experiments show a 20.7 percentage-point improvement over the strongest external baseline. In controlled policy-only settings, the pseudo-trained shot-level agent video encoder policy beats carefully hand-tuned policies while reducing system prompt usage by 74.3%. The authors release the full framework, an agent video reconstruction benchmark, and the first high-quality cinematic KG dataset.

Overview

  • Field: Computer Vision
  • Authors: Chuyue Li, Jinpeng Yu, Haozhe Wang
  • arXiv: 2508.03420
  • Key points

    Creative agents still lack an effective way to learn from high-quality human films, limiting their ability to produce cinematic-quality videos. A central challenge is the absence of a structured video representation that is both faithful to cinematic content and directly usable for agent reasoning and manipulation. The paper proposes the Agentic Video Auto-Encoder (AVA-Encoder), a framework for learning agent-native video representations via agent-based auto-encoding.

    Method

  • Videos are converted into knowledge graph (KG) representations, then reconstructed back into video.
  • Hierarchical and state nodes store structured text; a linked asset layer holds generated images, audio, and video.
  • Typed edges preserve the relationships between textual descriptions and assets in a form agents can easily understand, query, and edit.
  • Reconstruction error drives a text-gradient optimization framework: evaluation feedback is expressed as natural-language update directions, used for:
  • Outer loop: pseudo-training of a data-independent encoding strategy.
  • Inner loop: test-time refinement of the data-dependent KG representation.
  • Results

  • Improves over the strongest external baseline by 20.7 percentage points.
  • In a controlled policy-only setting, the pseudo-trained shot-level agent video encoder policy outperforms carefully hand-tuned policies while using 74.3% fewer system prompt tokens.

Releases

The authors release:

1. The complete AVA-Encoder framework. 2. A reliable agent video reconstruction benchmark. 3. The first high-quality cinematic KG representation dataset.

Tags

#computer-vision#video-representation#knowledge-graph#creative-agents#autoencoder#arxiv#video-generation#text-gradient-optimization

This page is an English static mirror generated for search and AI citation. It may be a full translation or structured summary of the Chinese original. Canonical interactive discussion lives on the Chinese page: https://zhichai.net/topic/178633452