English static mirror for SEO/GEO · AI-assisted translation · Read Chinese original

Searching Videos as Trees: Self-Correcting Agents for Grounded Long Video QA

Forum topic · 小凯 · 2026-07-21

Summary

VideoTreeSearch (VTS) is a framework for grounded long-video question answering (Grounded LVQA) that requires answering questions about long videos while localizing the short evidence intervals supporting the answers. Unlike prior agentic methods that rely on a single crop_video(start, end) action and often converge prematurely, VTS casts the task as iterative self-correcting search over an adaptive temporal tree. The tree is built from visual scene boundaries so each node corresponds to a semantically coherent segment, and an agent navigates it via four explicit operations: zoom_in, zoom_out, shift, and answer. Training combines a trajectory synthesis pipeline (including deliberate detours into wrong branches followed by recovery), supervised fine-tuning, and reinforcement learning with grounding and answer-accuracy rewards. On CG-Bench, Haystack-LVBench, and Haystack-Ego4D, VTS improves over the strongest prior agentic methods by +12.5 mIoU and +7.4 T-F1 respectively. The learned policy also transfers to general long-video QA, beating prior agentic baselines on Video-MME, MLVU, and LVBench by up to +7.1 accuracy points. arXiv: 2507.15489.

Paper Overview

Research area: cs.CV Authors: Ce Zhang, Ziyang Wang, Yulu Pan Published: 2026-07-21 arXiv: 2507.15489

Abstract

Grounded long-video question answering (Grounded LVQA) requires answering a question about a long video while localizing the short evidence interval that supports the answer. Recent agentic methods frame this task as multi-turn exploration with a single crop_video(start, end) action, which supports coarse-to-fine narrowing but provides no primitive for fine-to-coarse backtracking. As a result, these agents typically converge prematurely and cannot recover from an early mistake.

We propose VideoTreeSearch (VTS), a framework that casts grounded LVQA as iterative self-correcting search over an adaptive temporal tree. VTS constructs a non-uniform tree from visual scene boundaries so that each node corresponds to a semantically coherent segment, and trains an agent to navigate the tree through four discrete operations: zoom_in, zoom_out, shift, and answer. These operations expose backtracking and recovery as explicit, learnable primitives rather than implicit behaviors.

To train this navigation, we introduce a trajectory synthesis pipeline that produces multi-step paths through the tree, including deliberate detours into incorrect branches followed by recovery. We use these trajectories for supervised fine-tuning, followed by reinforcement learning with grounding and answer-accuracy rewards.

Key Results

  • Grounded LVQA benchmarks (CG-Bench, Haystack-LVBench, Haystack-Ego4D): VTS outperforms the strongest prior agentic methods by +12.5 mIoU on CG-Bench and +7.4 T-F1 on Haystack-Ego4D.
  • General long-video QA transfer: surpasses all prior agentic baselines on Video-MME, MLVU, and LVBench by up to +7.1 accuracy points.
  • Ablations confirm that self-correcting hierarchical search is the central mechanism behind these gains.
--- *Automatically collected on 2026-07-21.*

Tags

#long-video-qa#video-understanding#agentic-ai#reinforcement-learning#grounding#tree-search#computer-vision#arxiv-paper

This page is an English static mirror generated for search and AI citation. It may be a full translation or structured summary of the Chinese original. Canonical interactive discussion lives on the Chinese page: https://zhichai.net/topic/178446968