LeVJEPA: Video Pretraining with 1/20 the Compute via Gaussian Regularization (SIGReg)

Forum topic · 小凯 · 2026-08-28

Summary

LeVJEPA (arXiv:2608.27395) is a simplified video self-supervised pretraining framework from a team including Yann LeCun. It replaces V-JEPA's anti-collapse machinery — EMA target encoder, stop-gradient, capacity-limited predictor, and hand-crafted tube masks — with a single regularizer called SIGReg, which pushes representations toward an isotropic standard Gaussian via the Cramér-Wold theorem and the Epps-Pulley normality statistic, provably excluding collapse. The resulting architecture needs only an encoder plus a small projection head, trains with a fixed regularization weight (lambda = 0.02), tolerates uniformly dropping 95% of patch tokens (improving ImageNet-1K attentive probing accuracy from 33.9% to 47.6%), and supports block-causal attention with no accuracy loss, enabling streaming inference and future world models. Matched-FLOPs experiments show 5.6–20.8x training-cost reductions versus V-JEPA 2 across ViT-S/B/L, a 7.6-point gain over video baselines on ImageNet-1K, top accuracy on Kinetics-400, and emergent dense patch-level semantics. ViT-Tiny trained 12 hours on a single RTX 4090 achieves nontrivial ImageNet accuracy, making video pretraining accessible to small labs.

LeVJEPA: Video Pretraining with 1/20 the Compute via Gaussian Regularization

Paper: LeVJEPA: Efficient & Scalable Video Pretraining without the Heuristics Authors: Lukas Kuhn, Lucas Maes, Giuseppe Serra, Quentin Le Lidec, Yann LeCun, Randall Balestriero, Florian Buettner arXiv: 2608.27395

This post discusses LeVJEPA, which applies a "subtraction philosophy" to video pretraining: instead of a more complex model, a simpler one that works better.

Why video pretraining is hard

Time curse: a 10-second clip is 240–300 images, so compute costs are tens to hundreds of times higher than images, with an added causality/temporal-continuity problem.
Representation collapse: in joint-embedding self-supervised learning, models can map all inputs to a single vector (like answering "C" to every exam question), making predictions trivially "correct" while learning nothing.
V-JEPA's workarounds: EMA target encoder, stop-gradient, capacity-limited predictor, and carefully designed spatiotemporal tube masks all prevent collapse — but add hyperparameters and run three networks simultaneously.

The core idea: SIGReg

LeVJEPA's answer is SIGReg (Statistical Invariance Gaussian Regularization). If representations collapse, their distribution degenerates to zero variance. Forcing representations toward an isotropic standard Gaussian makes collapse impossible — and under mild assumptions, isotropic Gaussians minimize worst-case downstream probing risk.

Practically, via the Cramér–Wold theorem, high-dimensionality is handled by projecting representations onto randomly sampled directions and penalizing non-Gaussianity of each 1-D projection with the Epps–Pulley statistic. The loss is bounded, robust to outliers, needs no whitening or centering, and provably excludes collapse (collapsed directions have zero variance, maximally distant from unit variance).

What gets removed

| Component | V-JEPA / V-JEPA 2 | LeVJEPA | |---|---|---| | Anti-collapse | EMA encoder + stop-grad + limited predictor | Single SIGReg term | | Trainable architecture | Encoder + predictor + frozen target net | Encoder + small projection head | | Loss | Many hyperparameters | One fixed λ = 0.02 | | Attention | Fully bidirectional | Block-causal, no accuracy loss | | Token masking | Complex multi-block tube masks | Uniform random dropping (up to 95%) |

Efficiency results

5.6–20.8× training-cost reduction vs. V-JEPA 2 at matched epochs/data across ViT-S/B/L.
Token dropping helps: dropping 95% of patch tokens raises ImageNet-1K attentive probing accuracy from 33.9% to 47.6% — sparse observation acts as strong augmentation — while cutting feed-forward compute ~20×.
Block-causal attention enables streaming, constant per-frame incremental cost, and is a prerequisite for world models that predict the future.
Consumer GPU: ViT-Tiny trained 12 hours on one RTX 4090 with unlabeled walking videos achieves nontrivial ImageNet accuracy.

Benchmark highlights

Matched-FLOPs: beats all video baselines on ImageNet-1K by 7.6 points; approaches DINOv2 on appearance tasks, nearly doubles it on motion tasks.
Kinetics-400: highest accuracy among compared methods at matched FLOPs.
Something-Something-v2: heavy token dropping initially hurts motion performance but the gap closes with longer training; it remains the most compute-efficient setting.
Emergent dense representations: despite supervising only the [CLS] token, patch tokens self-organize into semantic structures (PCA cleanly separates foreground/background) — free compared to V-JEPA 2.1's explicit patch-level auxiliary loss.

Takeaway

V-JEPA's heuristics were patches around the collapse problem. LeVJEPA goes back to first principles: collapse is a loss of distributional diversity, so constrain the distribution's shape directly. The authors conclude that once the compute burden is removed, video becomes a viable — and in several respects preferable — substrate for general visual pretraining, since video is the native format of the physical world.

References

Kuhn, L., Maes, L., Serra, G., Le Lidec, Q., LeCun, Y., Balestriero, R., & Buettner, F. (2026). LeVJEPA: Efficient & Scalable Video Pretraining without the Heuristics. *arXiv preprint arXiv:2608.27395*.