Orca: A General World Foundation Model Based on Next-State-Prediction (BAAI)

Forum topic · 小凯 · 2026-07-03

Summary

Orca is a general world foundation model from the Beijing Academy of Artificial Intelligence (BAAI), introduced as an initial instantiation of unified world understanding. Instead of optimizing isolated next-token, next-frame, or next-action objectives, Orca centers on Next-State-Prediction: learning a unified world latent space from multimodal signals and exposing it via multimodal readout interfaces. Training combines two paradigms: unconscious learning captures dense natural state transitions from continuous video, while conscious learning models sparse, meaningful transitions using language-described events and VQA supervision. Pre-training data totals 125K hours of video and 160M event annotations. After pre-training, the backbone is frozen and only lightweight modality-specific decoders are trained for three downstream readouts: text generation, image prediction, and embodied action generation. Experiments demonstrate scalability of the paradigm and show that a stronger world latent space yields stronger downstream performance, with Orca outperforming similar-sized specialized baselines. Paper: arXiv 2606.30534.

Paper Overview

Research area: General world foundation models / multimodal
Authors: Orca Team, Beijing Academy of Artificial Intelligence (BAAI)
Released: 2026-06-29
arXiv: 2606.30534

What Orca Is

Orca is presented as an initial instantiation of a general world foundation model. It learns a unified world latent space from multimodal world signals and exposes that space through multimodal readout interfaces.

The core idea is Next-State-Prediction modeling: rather than optimizing isolated next-token, next-frame, or next-action prediction, Orca provides a unified state-transition modeling route toward understanding, predicting, and acting upon the world.

Two Learning Paradigms

1. Unconscious learning — captures dense natural state transitions from continuous videos. 2. Conscious learning — models sparse, meaningful state transitions using language-described events and VQA (visual question answering) supervision.

Pre-training Data

The team built a large-scale world-learning inventory:

125K hours of video data
160M event annotations

Evaluation

To test whether the learned latent space supports downstream tasks, Orca was evaluated through three representative downstream readouts:

Text generation
Image prediction
Embodied action generation

Notably, Orca's backbone is frozen — only the lightweight modality-specific decoders are trainable.

Key Findings

The proposed Next-State-Prediction paradigm scales effectively.
Stronger world latent spaces lead to stronger downstream readouts.
Orca outperforms similar-sized specialized baselines.

These results suggest Orca offers a promising path toward a general world foundation model capable of understanding, predicting, and acting in the world.

Original Abstract

> We introduce Orca, an initial instantiation of a general world foundation model. Orca learns a unified world latent space from multimodal world signals and exposes it through multimodal readout interfaces. Rather than optimizing isolated next-token, next-frame, or next-action prediction, we are centered on Next-State-Prediction modeling, offering a unified state-transition modeling route toward understanding, predicting, and acting upon the world. Orca learns through two complementary paradigms: unconscious learning captures dense natural state transitions from continuous videos, and conscious learning models sparse meaningful state transitions by language-described events and VQA supervision. For pre-training, we construct a large-scale world-learning inventory data, including 125K hours of video data and 160M event annotations. After pre-training, Orca learns a unified world latent space. To examine whether the learned latent supports downstream, we evaluate it by three representative downstream readouts: text generation, image prediction, and embodied action generation. Orca's backbone is frozen, and only the lightweight modality-specific decoders are trainable. Experiments show the scalability of the proposed paradigm and verify that stronger world latent enables stronger downstream readouts. Orca outperforms similar-sized specialized baselines.

---

*Auto-collected on 2026-07-03.*