S³T: Temporal Self-Distillation for Visual State Tracking in Videos

Forum topic · 小凯 · 2026-09-05

Summary

Researchers Shravan Venkatraman, Wenshuai Zhao, and Mohammad Hassan Vali introduce S³T (Self-Supervised Self-Distillation over Time), presented as the first fully self-contained framework for continuous video state tracking in computer vision. The method treats temporal sampling density as privileged information: a densely sampled view of the same video clip serves as a teacher, while a sparse-view student with shared weights learns to match the teacher's next-token distribution. Because the model generates its own targets, training requires no labels, no separate teacher model, and no reward signal, and adds zero inference cost. Applied to LLaVA-OneVision-2-8B, S³T improves VSTAT accuracy by +1.74 as a single model, +2.38 with model merging, and +2.70 with visual encoder adaptation, whereas prior self-evolution methods showed little state-tracking improvement. Capabilities learned from unlabeled synthetic videos transfer to real videos, yielding +7.95 on VSTAT-YouTube state tracking and +4.50 on MVBench action counting. The paper is available on arXiv (2609.04203).

Paper Overview

Field: CV Authors: Shravan Venkatraman, Wenshuai Zhao, Mohammad Hassan Vali Published: 2026-09-03 arXiv: 2609.04203

Abstract

We introduce S $$^3$$ T (Self-Supervised Self-Distillation over Time), which is the first fully self-contained framework for continuous video state tracking. Our method treats temporal sampling density as privileged information based on the hypothesis that a denser view of the same clip recovers the running state more accurately. This view serves as the teacher, while a sparse-view student with the same weights learns to match its next-token distribution. The model generates its own target, so training requires no labels, separate teacher, or reward signal, and adds no inference cost.

Key Results

On LLaVA-OneVision-2-8B, S³T as a single model improves VSTAT accuracy by +1.74.
With model merging, the improvement reaches +2.38; adding visual encoder adaptation brings it to +2.70.
Prior self-evolution methods show almost no improvement on state tracking.
Capabilities learned from unlabeled synthetic videos transfer to real videos: +7.95 on VSTAT-YouTube state tracking questions and +4.50 on MVBench action counting.

Core Idea

Temporal sampling density is used as privileged information: denser sampling of the same clip recovers the running state more accurately.
The dense view acts as the teacher; a sparse-view student with shared weights matches the teacher's next-token distribution.
Fully self-contained training: no labels, no separate teacher model, no reward signal, and no extra inference cost.

--- *Auto-collected on 2026-09-05*