Paper Overview
Field: Computer Vision Authors: Adrien Ramanana Rahary, Yutong Bai, Nicolas Dufour, Patrick Perez, David Picard Published: 2026-10-21 arXiv: 2610.30129
Summary
Video world models predict future frames from past frames and actions. They serve as simulators in which agents train, plan or are evaluated, and that humans can control interactively. They generate in the latent space of a video autoencoder, whose compression sets their cost per frame and how long each action waits to take effect. Yet world models often reuse the autoencoders of text-to-video generators, built for offline generation with no agent waiting and, without their training data and code, not retrainable at another compression. We therefore ask how, where and how much a world model should compress.
We build CADRE, an open family of causal diffusion representation autoencoders spanning multiple spatio-temporal compressions. Its members differ only in a small bottleneck on frozen features, and share a diffusion decoder self-distilled to one or few steps. We train identical world models on two manipulation and two navigation environments. CADRE constitutes most of the latency-quality frontier, especially in scenes with camera motion.
Key Findings
- Under strong compression, diffusion decoders produce sharper results than perceptual regression.
- In robotic manipulation, temporal compression is more effective inside the autoencoder than inside the world model.
- Manipulation and navigation call for opposite compression strategies: temporal compression is nearly free when the camera is static, while spatial compression is preferable when the camera moves.
- Surprisingly, with a static camera, coarser spatial tokens hurt motion far more than they hurt image quality.
*Auto-collected on 2026-10-23*
#paper #arXiv #CV