ALPS: Visual State Encoders for World Models Unify Prediction, Language, and Reconstruction

Forum topic · 小凯 · 2026-10-23

Summary

ALPS is a pretrained visual state encoder designed to serve as a unified visual foundation for world models. It combines three capabilities—physical-state information, native language alignment, and pixel reconstruction—within a single set of patch tokens, whereas existing encoders typically specialize in only one or two. The authors pretrain a ViT-B/16 from scratch using only still images and text, with language, reconstruction, and spatial correspondence objectives, then refine it via cross-view self-distillation. In a unified evaluation across fifteen public encoders, ALPS achieves the lowest aggregate LIBERO state-probing error among ViT-B models, matching V-JEPA 2-L, which is 3.5x larger, while maintaining strong language alignment and accurate pixel reconstruction. Downstream, frozen ALPS tokens with a native text tower reach up to 85.5% average policy success rate on LIBERO, match or exceed leading encoders in world-model planning on PushT, Wall, and Granular, and allow predicted future tokens to be decoded directly into pixels for visual inspection. Paper: arXiv 2610.30120.

Paper Overview

Research area: Computer Vision Authors: Yanqing Liu, Letian Zhang, Chengzhi Liu, Zijun Wang, Jieru Mei, Hongliang Fei, Yuyin Zhou, Cihang Xie arXiv: 2610.30120

Key Points

World models turn visual representations into predictions and actions. The authors argue that a world model's visual foundation should combine three capabilities: physical-state information, native language alignment, and pixel reconstruction—so a single representation supports prediction, instruction following, and visual inspection.
Existing encoders typically specialize in only one or two of these capabilities. ALPS is introduced as a pretrained visual state encoder that unifies all three in a single set of patch tokens.
Training uses only still images and text: a ViT-B/16 is pretrained from scratch with language, reconstruction, and spatial correspondence objectives, then refined via cross-view self-distillation.
In a unified evaluation across fifteen public encoders, ALPS achieves the lowest aggregate LIBERO state-probing error among ViT-B models, on par with V-JEPA 2-L (3.5x larger), while retaining strong language alignment and accurate pixel reconstruction.
Downstream results: frozen ALPS tokens paired with a native text tower achieve the highest average policy success rate on LIBERO (85.5%); ALPS matches or exceeds leading encoders in world-model planning on PushT, Wall, and Granular; and predicted future tokens can be decoded directly into pixels for visual inspection.

Original Abstract (excerpt)

> World models turn visual representations into predictions and actions. We argue that their visual foundation should combine physical-state information, native language alignment, and pixel reconstruction, so that a single representation can support prediction, instruction, and inspection. However, existing encoders typically specialize in only one or two of these capabilities; we therefore introduce ALPS, a pretrained visual state encoder that unifies all three in a single set of patch tokens. Using only still images and text, we pretrain a ViT-B/16 from scratch with language, reconstruction, and spatial correspondence objectives, and then refine it via cross-view self-distillation. In a unified evaluation across fifteen public encoders, ALPS achieves the lowest aggregate LIBERO state-prob...

Paper link: https://arxiv.org/abs/2610.30120