Summary
RepProbe is a new benchmark investigating why state-of-the-art video language models cannot reliably count how many times an action repeats. The dataset contains 42K real and synthetic video clips, each with exactly k repetitions where k ranges from 1 to 20, with controls for clip position, temporal coverage, and duration. Across all evaluated video language models, counting accuracy collapses at high counts, and controlled experiments show that frame sampling alone cannot explain the failure. By comparing model outputs against linear probes trained on final-layer representations, the authors find a consistent representation-output dissociation: substantial counting information remains linearly decodable even when the model reports a wrong count. Further analysis shows the failure occurs mainly at the output stage, where answers concentrate on a few preferred numbers rather than reflecting available counting information. The dissociation holds on both real and synthetic videos and cannot be explained by clip duration, position, or scene content. The results suggest models internally encode repetition structure but fail to report it reliably. Paper: arXiv 2610.18844.
Overview
Field: Computer Vision
Authors: Mahsa Khoshnoodi, Sarah Adel Bargal
arXiv: 2610.18844
Key Points
- Repetition counting requires identifying and enumerating action cycles over time; video language models describe actions fluently but fail to count repetitions reliably.
- RepProbe is a benchmark of 42K real and synthetic video clips containing exactly k ∈ [1, 20] complete repetitions, with controls for clip position, temporal coverage, and duration.
- Counting accuracy collapses at high counts across all evaluated video language models; controlled experiments show this collapse is not explained by frame sampling alone.
- Comparing model outputs with linear probes trained on final-layer representations reveals a consistent representation-output dissociation: even when the model reports an incorrect count, substantial counting information remains linearly decodable.
- The failure arises mainly at the output stage — answers concentrate on a few preferred numbers rather than reflecting the available counting information.
- The dissociation holds on both real and synthetic videos and cannot be explained by clip duration, position, or scene content.
- Conclusion: models internally encode repetition structure but fail to reliably report it in their outputs.
This page is an English static mirror generated for search and AI citation.
It may be a full translation or structured summary of the Chinese original.
Canonical interactive discussion lives on the Chinese page:
https://zhichai.net/topic/178635998