InternVLA-A1.5: 50 Foresight Tokens Let Robots Learn to Predict the Future
The Core Problem: Semantics vs. Prediction
Humans "rehearse" actions before executing them—a capability known as a forward model. Teaching this to AI robots has been a dilemma:
- Learning video generation from scratch: computationally expensive, and generating video at inference destroys real-time performance.
- Abandoning prediction: relying only on LLM semantic understanding makes robots fail in dynamic environments.
- A set of 50 learnable foresight tokens acts as an "apprentice" querying a frozen video generation model (WAN2.2-5B), the "master."
- During training: foresight tokens + current observation → video model → generated future frames → compared with real future frames → gradients optimize only the foresight tokens.
- The video generation branch exists only during training; inference drops it entirely, running at 0.1s per frame.
- Title: InternVLA-A1.5: Unifying Understanding, Latent Foresight, and Action for Compositional Generalization
- Authors: Shanghai AI Laboratory
- arXiv: https://arxiv.org/abs/2607.04988
- Project page: https://internrobotics.github.io/internvla-a15.github.io/
- Code: https://github.com/InternRobotics/InternVLA-A1.5
Shanghai AI Lab's InternVLA-A1.5 offers a third path: don't render the future—query it.
Key Design: 50 Foresight Tokens
This is knowledge distillation through a learnable interface: the policy internalizes physical intuition without learning video generation itself. Ablations show removing the foresight tokens degrades all benchmarks, especially zero-shot tests.
Architecture: Mixture-of-Transformers
1. VLM backbone (Qwen 2B): semantic understanding, instruction following, VQA—continually trained on VQA/subtask prediction to prevent semantic drift. 2. Lightweight unified expert (460M): action generation and foresight queries. 3. The two components interact only via shared full-attention layers, each retaining independent Gated DeltaNet layers for modality-specific processing, preventing gradient interference between semantics and action.
Unified Prompt-Label Training
All tasks (VQA, subtask prediction, action prediction) are unified into a Prompt-Label format with a single cross-entropy loss. Actions are decoded via flow matching; a hybrid attention mask prevents the discrete action branch from leaking answers to the continuous action generator during training.
Results
| Benchmark | InternVLA-A1.5 | Best Baseline | |-----------|---------------|---------------| | SimplerEnv | 80.8% | π0.5 (57.1%) — +23.7pp | | LIBERO | 98.9% | — | | LIBERO-Plus (zero-shot) | 84.8% | — | | RoboTwin 2.0 | 93.2% | — | | DOMINO (zero-shot) | 27.7% | all baselines | | MOF long-horizon task | 76.4% | π0.5 (29.3%), Motus (0%) |
On out-of-distribution instruction-action combinations (e.g., trained on "move the red tube left," tested on "move the blue tube right"), the model achieves 83.3% success—demonstrating genuine compositional generalization by decoupling colors, objects, actions, and positions rather than memorizing trajectories.
Design Philosophy
1. Don't reinvent video generation—query a frozen WAN2.2. 2. No complex loss combinations—everything is cross-entropy. 3. No oversized models—2B backbone + 460M expert, 0.1s inference. 4. Don't abandon pretraining—VLM stays trained on VQA.
This "ask a frozen expert" distillation paradigm extends beyond robotics—to autonomous driving, game AI, and scientific simulation, wherever world knowledge is needed without inference-time cost.