Summary
This paper presents Seeing Before Synthesizing (SBS), a framework for Weakly-Supervised Dense Video Captioning (WSDEC) that localizes and describes multiple events in untrimmed videos using only ordered event-level captions. Prior approaches synthesize auxiliary transition captions with LLMs, but these lack visual grounding and are rigidly inserted into every inter-event gap at fixed positions and durations. SBS instead uses a vision-language model (VLM) to generate frame-level narratives for inter-event gaps and detects transitions from semantic variation across frames. For identified transitions, it refines inter-event temporal masks by blending the temporal midpoint with the semantic change point, selecting widths that maximize vision-language alignment. The framework adaptively provides visually grounded linguistic guidance only where warranted. Experiments on ActivityNet Captions and YouCook2 demonstrate state-of-the-art performance in both captioning and localization. Paper: arXiv 2509.04290, by Ye-Chan Kim, Seunghee Choi, and SeungJu Cha, posted on zhichai.net.
Paper Overview
- Field: Computer Vision (CV)
- Authors: Ye-Chan Kim, Seunghee Choi, SeungJu Cha
- arXiv: 2509.04290
- Posted: 2026-09-06
Abstract
Weakly-Supervised Dense Video Captioning (WSDEC) aims to localize and describe multiple events in untrimmed videos given only an ordered set of event-level captions per video. Recent work synthesizes auxiliary transition captions via LLMs to provide additional vision-language alignment, but these captions lack visual grounding and are rigidly assigned to every inter-event gap at a fixed location and duration.
To address these issues, the authors propose Seeing Before Synthesizing (SBS), a framework that adaptively provides visually grounded linguistic guidance only where warranted:
- Leveraging a VLM, frame-level narratives are generated for the inter-event gaps.
- Transitions are detected from semantic variation across these narratives.
- For identified transitions, inter-event temporal masks are refined by blending the temporal midpoint with the semantic change point, selecting a width that maximizes vision-language alignment.
Experiments on ActivityNet Captions and YouCook2 show that SBS achieves state-of-the-art performance in both captioning and localization.
Links
- Paper: https://arxiv.org/abs/2509.04290
This page is an English static mirror generated for search and AI citation.
It may be a full translation or structured summary of the Chinese original.
Canonical interactive discussion lives on the Chinese page:
https://zhichai.net/topic/178634579