Reviso: An Agentic Video Captioning Framework That Prioritizes Key Visual Evidence

Forum topic · 小凯 · 2026-11-03

Summary

Reviso is a new agentic video captioning framework introduced in an arXiv paper (2611.03602) from researchers including Tengfei Liu, Yang Shi, and Pengfei Wan. The work argues that multimodal large language models (MLLMs) often overemphasize trivial visual details in video captioning while missing essential context and key events, since not all visual evidence deserves equal attention. Reviso learns what to observe and how to gather relevant evidence through a hierarchical Context-Focus-Points (CFP) observation plan built from a low-frame-rate global preview. It then performs goal-directed active perception, selectively revisiting temporal segments and combining tools for fine-grained inspection, updating its plan as needed. Training uses a verification-guided pipeline with fact checking and teacher-guided refinement, plus GRPO with a critical-fact reward, plan-alignment reward (PAR), and unproductive-tool-call penalty (UTCP). Reviso outperforms evaluated open-source baselines on four video captioning benchmarks and produces captions that better support downstream video question answering.

Paper Overview

Field: Computer Vision (CV)
Authors: Tengfei Liu, Yang Shi, Yuran Wang, Yuqi Tang, Yizhuo Jia, Zhuoran Zhang, Qixun Wang, Shujie Li, Xuanyu Zhu, Xinyu Liu, Minghui Guo, Shixuan Liu, Haotian Wang, Yuanxing Zhang, Pengfei Wan
arXiv: 2611.03602

Abstract (translated from Chinese summary)

Multimodal large language models (MLLMs) can generate detailed video captions, yet they may overemphasize minor details while overlooking essential context and key events. The authors argue that not all visual evidence deserves equal attention — the importance of visual information depends on the content of each video.

Based on this observation, they propose Reviso, an agentic video captioning framework that learns *what to observe* and *how to gather relevant visual evidence*:

1. Hierarchical observation planning: Starting from a low-frame-rate global preview, Reviso builds a Context-Focus-Points (CFP) plan that identifies the video context, prioritizes key aspects, and defines concrete observation targets with corresponding tools. 2. Goal-directed active perception: Guided by the plan, Reviso selectively revisits temporal segments and combines tools for fine-grained inspection. It interprets gathered evidence, updates the plan as needed, and integrates global and local information to produce the final caption. 3. Verification-guided training: High-quality training trajectories are constructed via factual verification and teacher-guided refinement. 4. GRPO-based RL training with three rewards:

a critical-fact reward for caption quality,
a Plan-Alignment Reward (PAR) for identifying important observation targets,
an Unproductive Tool-Call Penalty (UTCP) that modulates observation cost based on caption quality.

Results

Reviso outperforms all evaluated open-source baselines on four video captioning benchmarks.
Its captions more effectively support downstream video question answering.
The findings highlight the effectiveness of priority-guided active observation in video captioning.

Original Abstract (excerpt)

> Multimodal large language models (MLLMs) can generate detailed video captions, yet they may overemphasize minor details while overlooking essential context and key events. We argue that not all visual evidence deserves equal attention, as the importance of visual information depends on the content of each video. Motivated by this observation, we propose Reviso, an agentic video captioning framework that learns what to observe and how to gather relevant visual evidence. From a low-frame-rate global preview, Reviso constructs a hierarchical Context--Focus--Points (CFP) observation plan...

*Auto-collected on 2026-11-04.*