VideoMSN: Image Classifiers as Efficient Self-Supervised Video Representation Learners

Forum topic · 小凯 · 2026-10-02

Summary

VideoMSN is a Masked Siamese Network framework for efficient self-supervised spatio-temporal video representation learning, introduced by Owais Iqbal, Sudipta Sarkar, and Shyam Marjit (arXiv:2609.26768). Instead of relying on heavy 3D architectures or reconstruction-based autoencoders, VideoMSN repurposes standard image Vision Transformers by representing videos as super images — grids composed of frames sampled from videos. Two views are constructed from each super image: one with spatial patch masking and the other with temporal frame masking, ensuring no information leakage across frames. A shared ViT encoder aligns their embeddings with a masked Siamese loss, capturing motion and appearance cues without reconstruction. This decoder-free approach leverages image foundation models for efficient video learning. Starting from pretrained DINO-v3 and DeiT-v3 encoders, VideoMSN achieves state-of-the-art results on Kinetics-400, UCF101, and HMDB51 while reducing video pretraining epochs by up to 32x and 160x versus prior video SSL methods, and shows strong performance in low-sample classification, confirming transferability in label-scarce settings.

Paper Overview

Field: Computer Vision Authors: Owais Iqbal, Sudipta Sarkar, Shyam Marjit Published: 2026-09-30 arXiv: 2609.26768

Abstract

We introduce VideoMSN, a Masked Siamese Network framework for efficient self-supervised spatio-temporal representation learning in videos. Instead of relying on heavy 3D architectures or reconstruction-based autoencoders for learning with unlabeled data, we repurpose standard image Vision Transformers by representing videos as super images — grids composed of frames sampled from videos. From each super image, we construct two views: one with spatial patch masking and the other with temporal frame masking, ensuring no information leakage across frames. A shared Vision Transformer (ViT) encoder aligns their embeddings using a masked Siamese loss, capturing both motion and appearance cues without reconstruction. Our decoder-free formulation leverages an image foundation model towards efficient video representation learning.

Key Results

Starting from pretrained DINO-v3 and DeiT-v3 image encoders, VideoMSN achieves state-of-the-art performance on Kinetics-400, UCF101, and HMDB51.
Compared with prior video self-supervised learning methods, it reduces video pretraining epochs by up to 32x and 160x.
The method also excels in low-sample classification, confirming the transferability of learned representations in label-scarce scenarios.

Method Highlights

Videos are represented as super images: grids composed of frames sampled from the video.
Two views are built per super image — one with spatial patch masking, one with temporal frame masking — ensuring no information leakage across frames.
A shared ViT encoder aligns the view embeddings with a masked Siamese loss, capturing both motion and appearance cues without any reconstruction step.
The decoder-free design makes direct use of image foundation models for efficient video learning.

---

*Auto-collected on 2026-10-02*