[论文] Searching Videos as Trees: Self-Correcting Agents for Grounded Long Video QA

论文概要

研究领域: cs.CV
作者: Ce Zhang, Ziyang Wang, Yulu Pan
发布时间: 2026-07-21
arXiv: 2507.15489

中文摘要

有根据的长视频问答（Grounded LVQA）要求在回答关于长视频的问题时，同时定位支持答案的短证据区间。近期基于智能体的方法将这一任务框架为单步crop_video(start, end)动作的多轮探索，支持由粗到细的缩小但不提供由细到粗的回溯原语。因此，这些智能体通常会过早收敛，无法从早期错误中恢复。我们提出VideoTreeSearch（VTS），一个将有根据的LVQA转化为自适应时间树上的迭代自校正搜索的框架。VTS从视觉场景边界构建非均匀树，使每个节点对应语义一致的片段，并训练智能体通过四种离散操作在树上导航：zoom_in、zoom_out、shift和answer。这些操作将回溯和恢复作为显式的可学习原语，而非隐式行为。为训练这种导航，我们引入轨迹合成流程，生成穿过树的多步路径，包括故意进入错误分支后的恢复。我们使用这些轨迹进行监督微调，随后使用基于定位和答案准确性的奖励进行强化学习。在三个Grounded LVQA基准（CG-Bench、Haystack-LVBench、Haystack-Ego4D）上，VTS比最强的先前智能体方法在CG-Bench上提升+12.5 mIoU，在Haystack-Ego4D上提升+7.4 T-F1。学到的策略还能迁移到一般长视频QA，在Video-MME、MLVU和LVBench上超越所有先前智能体基线达+7.1准确率点。消融实验确认自校正分层搜索是这些收益的核心机制。

原文摘要

Grounded long-video question answering (Grounded LVQA) requires answering a question about a long video while localizing the short evidence interval that supports the answer. Recent agentic methods frame this task as multi-turn exploration with a single crop_video(start, end) action, which supports coarse-to-fine narrowing but provides no primitive for fine-to-coarse backtracking. As a result, these agents typically converge prematurely and cannot recover from an early mistake. We propose VideoTreeSearch (VTS), a framework that casts grounded LVQA as iterative self-correcting search over an adaptive temporal tree. VTS constructs a non-uniform tree from visual scene boundaries so that each node corresponds to a semantically coherent segment, and trains an agent to navigate the tree through four discrete operations: zoom_in, zoom_out, shift, and answer. These operations expose backtracking and recovery as explicit, learnable primitives rather than implicit behaviors. To train this navigation, we introduce a trajectory synthesis pipeline that produces multi-step paths through the tree, including deliberate detours into incorrect branches followed by recovery. We use these trajectories for supervised fine-tuning, followed by reinforcement learning with grounding and answer-accuracy rewards. On three Grounded LVQA benchmarks (CG-Bench, Haystack-LVBench, Haystack-Ego4D), VTS outperforms the strongest prior agentic methods by +12.5 mIoU on CG-Bench and +7.4 T-F1 on Haystack-Ego4D. The learned policy also transfers to general long-video QA, surpassing all prior agentic baselines on Video-MME, MLVU, and LVBench by up to +7.1 accuracy points. Ablations confirm that self-correcting hierarchical search is the central mechanism behind these gains.

自动采集于 2026-07-21

#论文 #arXiv #CV #小凯