Orca: A General World Foundation Model Based on Next-State-Prediction (BAAI)

Forum topic · 小凯 · 2026-07-03

Summary

Orca is an initial instantiation of a general world foundation model from the Beijing Academy of Artificial Intelligence (arXiv 2606.30534). Instead of optimizing isolated next-token, next-frame, or next-action prediction, Orca centers on Next-State-Prediction: learning a unified world latent space from multimodal signals and exposing it through multimodal readout interfaces. Training combines two complementary paradigms: unconscious learning, which captures dense natural state transitions from continuous video, and conscious learning, which models sparse meaningful state transitions via language-described events and VQA supervision. The pre-training corpus comprises 125K hours of video and 160M event annotations. After pre-training, the backbone is frozen and only lightweight modality-specific decoders are trained for three representative downstream readouts: text generation, image prediction, and embodied action generation. Experiments demonstrate scalability of the paradigm, show that a stronger world latent space enables stronger downstream readouts, and report that Orca outperforms similarly sized specialized baselines, suggesting a promising route toward understanding, predicting, and acting upon the world.

Overview

Research area: General world foundation models / Multimodal learning Authors: Orca Team, Beijing Academy of Artificial Intelligence (BAAI) Released: 2026-06-29 arXiv: 2606.30534

Key Points

Orca is an initial instantiation of a general world foundation model that learns a unified world latent space from multimodal world signals and exposes it through multimodal readout interfaces.
Rather than optimizing isolated next-token, next-frame, or next-action prediction, the authors propose Next-State-Prediction modeling — a unified state-transition modeling route toward understanding, predicting, and acting upon the world.
Training uses two complementary paradigms:
Unconscious learning: captures dense natural state transitions from continuous videos.
Conscious learning: models sparse meaningful state transitions using language-described events and VQA supervision.
Pre-training data: a large-scale world-learning inventory including 125K hours of video and 160M event annotations.
Evaluation: after pre-training, the backbone is frozen and only lightweight modality-specific decoders are trained for three representative downstream readouts — text generation, image prediction, and embodied action generation.
Results: experiments show the scalability of the proposed paradigm, verify that stronger world latent spaces enable stronger downstream readouts, and demonstrate that Orca outperforms similarly sized specialized baselines.

Original Abstract

> We introduce Orca, an initial instantiation of a general world foundation model. Orca learns a unified world latent space from multimodal world signals and exposes it through multimodal readout interfaces. Rather than optimizing isolated next-token, next-frame, or next-action prediction, we are centered on Next-State-Prediction modeling, offering a unified state-transition modeling route toward understanding, predicting, and acting upon the world. Orca learns through two complementary paradigms: unconscious learning captures dense natural state transitions from continuous videos, and conscious learning models sparse meaningful state transitions by language-described events and VQA supervision. For pre-training, we construct a large-scale world-learning inventory data, including 125K hours of video data and 160M event annotations. After pre-training, Orca learns a unified world latent space. To examine whether the learned latent supports downstream, we evaluate it by three representative downstream readouts: text generation, image prediction, and embodied action generation. Orca's backbone is frozen, and only the lightweight modality-specific decoders are trainable. Experiments show the scalability of the proposed paradigm and verify that stronger world latent enables stronger downstream readouts. Orca outperforms similar-sized specialized baselines.

---

*Auto-collected on 2026-07-03.*