PercepCap: Video Captioner with Structured Spatio-Temporal Perception

Forum topic · 小凯 · 2026-07-24

Summary

PercepCap is a perception-aware video captioning framework presented in arXiv paper 2507.18394 by Yifan Xu, Zihao Wang, and Zhixiao Wang. Unlike existing multimodal LLMs that generate captions directly from video without exposing perceptual evidence, PercepCap follows a perceive-describe generation chain: the model first produces a spatio-temporal perception trace capturing object locations (boxes) and event timings (timestamps), then generates the final caption grounded in this evidence. Training uses a two-stage strategy: perception-then-describe supervised fine-tuning to adapt the model to the new chain, and perception-grounded reinforcement learning that jointly optimizes perception traces and caption quality via combined rewards. A caption-anchored perception data construction pipeline builds training ground truth by generating caption-only descriptions, extracting mentioned objects and events, and grounding them back into videos with boxes and timestamps. PercepCap consistently outperforms the Qwen3-VL baseline on direct captioning and caption-to-QA evaluations.

Overview

Research area: Computer Vision Authors: Yifan Xu, Zihao Wang, Zhixiao Wang Published: 2026-07-24 arXiv: 2507.18394

Abstract

Video captioning requires fine-grained spatio-temporal understanding of videos, including spatial perception of where objects are located and temporal perception of when events occur. Existing MLLMs usually generate captions directly from video inputs without exposing the perceptual evidence behind descriptions. As a result, mistakes in spatio-temporal perception are only observed in the final caption, making it difficult to identify the underlying perceptual errors directly.

To address these issues, the authors present PercepCap, a perception-aware video captioning framework that makes perceptual evidence explicit before producing the final caption. PercepCap follows a perceive-describe generation chain, where the model first produces a spatio-temporal perception trace comprising object trajectories and temporal events, then generates the final caption based on this perceptual evidence.

Key Components

Two-stage training strategy:
*Perception-then-describe supervised fine-tuning (SFT)*: adapts the model from caption-only generation to the proposed perceive-describe chain.
*Perception-grounded reinforcement learning (RL)*: optimizes perception trace and caption quality jointly via rewards on both the perception chain and the final caption.
Caption-anchored perception data construction: a pipeline that builds SFT and RL training data by first generating caption-only descriptions, extracting the objects and events mentioned in them, and grounding them back into the video with boxes and timestamps. This produces caption-aligned perception data providing reliable ground truth, ensuring that explicit perception traces and the final caption reference the same objects and events.

Results

On direct captioning and caption-to-QA evaluations, PercepCap consistently outperforms the Qwen3-VL baseline and demonstrates leading captioning quality.