PercepCap: A Video Captioner with Structured Spatio-Temporal Perception

Forum topic · 小凯 · 2026-07-24

Summary

PercepCap (arXiv:2507.18394) is a perception-aware video captioning framework by Yifan Xu, Zihao Wang, and Zhixiao Wang. It addresses a key limitation of existing multimodal large language models (MLLMs), which generate video captions directly without exposing the perceptual evidence behind them, making spatio-temporal errors hard to diagnose. PercepCap follows a perceive-then-describe generation chain: the model first produces a spatio-temporal perception trace containing object trajectories and temporal events, then generates the final caption grounded in that evidence. Training uses a two-stage strategy: perception-then-describe supervised fine-tuning, followed by perception-grounded reinforcement learning with joint rewards on the perception chain and final caption. A caption-anchored perception data construction pipeline builds training ground truth by generating captions, extracting mentioned objects and events, and grounding them back into the video with boxes and timestamps. PercepCap consistently outperforms a Qwen3-VL baseline on direct captioning and caption-to-QA evaluations.

Overview

Field: Computer Vision (CV) Authors: Yifan Xu, Zihao Wang, Zhixiao Wang Published: 2026-07-24 arXiv: 2507.18394

Abstract (Translation)

Video captioning requires fine-grained spatio-temporal understanding of videos, including spatial perception of where objects are located and temporal perception of when events occur. Existing MLLMs usually generate captions directly from video inputs without exposing the perceptual evidence behind descriptions. As a result, mistakes in spatio-temporal perception are only observed in the final caption, making it difficult to identify the underlying perceptual errors directly.

To address these issues, the authors present PercepCap, a perception-aware video captioning framework that makes perceptual evidence explicit before producing the final caption. PercepCap follows a perceive-describe generation chain, where the model first produces a spatio-temporal perception trace comprising object trajectories and temporal events, then generates the final caption based on the perceptual evidence.

Key Contributions

Two-stage training strategy:
*Perception-then-describe supervised fine-tuning (SFT)*: adapts the model from caption-only generation to the proposed perceive-describe chain.
*Perception-grounded reinforcement learning (RL)*: optimizes both the perception trace and caption quality via a joint reward on the perception chain and the final caption.
Caption-anchored perception data construction: a pipeline that first generates caption-only descriptions, extracts the objects and events mentioned, and grounds them back into the video with bounding boxes and timestamps. This yields caption-aligned perception data providing reliable training ground truth, ensuring the explicit perception traces and the final caption refer to the same objects and events.

Results

In both direct captioning and caption-to-QA evaluations, PercepCap consistently outperforms the Qwen3-VL baseline and demonstrates leading captioning quality.

---

*Auto-collected on 2026-07-24.*