AVA-Encoder: Agent-Native Video Representation Learning via Self-Encoding

Forum topic · 小凯 · 2026-08-14

Summary

This paper introduces Agentic Video Auto-Encoder (AVA-Encoder), a framework that converts video into knowledge-graph (KG) representations and then reconstructs it back to video. The goal is to give creative agents a structured representation faithful to cinematographic content and directly usable for reasoning and manipulation. The KG uses hierarchical and state nodes storing structured text, a linked asset layer holding generated images, audio, and video, and typed edges that preserve relationships between descriptions and assets in a form agents can query and edit. A text-gradient optimization framework driven by video-reconstruction differences expresses evaluation feedback as natural-language update directions, supporting both an outer-loop data-independent pseudo-training encoding policy and an inner-loop test-time data-dependent KG refinement. Experiments report a 20.7-point gain over the strongest external baseline and a 74.3% reduction in system-prompt usage, while outperforming hand-tuned policies. Authors release the framework, a reliable agentic video reconstruction benchmark, and the first high-quality cinematic KG dataset.

Overview

Field: Computer Vision (CV)
Authors: Chuyue Li, Jinpeng Yu, Haozhe Wang
Released: 2026-08-13
arXiv: 2508.03420

Problem

Creative agents still lack an effective way to learn from high-quality human films, limiting their ability to produce cinematic-level video. A key bottleneck is the absence of a structured video representation that is both faithful to the cinematic content and directly usable by agents for reasoning and manipulation.

Method

The paper proposes Agentic Video Auto-Encoder (AVA-Encoder), a framework that learns agent-native video representations through agentic self-encoding:

1. Encode video into a knowledge graph (KG)

Hierarchical and state nodes store structured text.
A linked asset layer stores generated images, audio, and video.
Typed edges preserve relationships between text descriptions and assets in a form that agents can easily query and edit.

2. Reconstruct video from the KG

Video reconstruction differences drive a text-gradient optimization framework.
Evaluation feedback is expressed as natural-language update directions.

3. Two operating loops

Outer loop: data-independent encoding-policy pseudo-training.
Inner loop: test-time, data-dependent refinement of the KG representation.

Results

Outperforms the strongest external baseline by 20.7 percentage points.
In a controlled policy-only setting, the pseudo-trained shot-level agentic video encoder policy also outperforms carefully hand-tuned policies.
Reduces system-prompt usage by 74.3%.

Contributions

Full release of the AVA-Encoder framework.
A reliable benchmark for agentic video reconstruction.
The first high-quality cinematic KG representation dataset.

Source

arXiv: https://arxiv.org/abs/2508.03420