OmniStream: Mastering Perception, Reconstruction and Action in Continuous Streams
> Research date: 2025-05-11 | arXiv: https://arxiv.org/abs/2603.12265 > Authors: Yibin Yan, Jilan Xu, Shangzhe Di, Haoning Wu, Weidi Xie > Institutions: School of AI, Shanghai Jiao Tong University; Shanghai Innovation Institute; Oxford VGG
Key points
- Motivation: Current vision foundation models are fragmented — specialized in image semantics (DINOv3), offline temporal modeling (V-JEPA 2), or spatial geometry (CUT3R). No single model handles semantics + space + time together, which is critical for real-time embodied agents.
- Core claim: A single visual backbone, kept strictly frozen, can be deployed across 29 different tasks without any fine-tuning.
- Frame *t* only attends to frames ≤ t; per-frame online processing with cached keys/values.
- Benchmarks (T=64, 224×224, single H800):
- Extends RoPE to height, width, and time axes.
- Enables zero-shot length extrapolation: trained on T=16 frames, infers on 110 frames (7× extrapolation).
- Joint pretraining on 29 datasets covering:
- Static + temporal representation learning (ImageNet, K400, SSv2, DAVIS'17)
- Streaming geometric reconstruction (Sintel, BONN, KITTI, TUM, ScanNet)
- Vision-language alignment (VSI-Bench, OVO-Bench)
- Backbone initialized from DINOv3; dual DPT heads predict depth/ray maps and camera pose; lightweight autoregressive language decoder links visual tokens to language.
- Video understanding: SSv2 68.5% (vs. DINOv3 54.0%), K400 74.2% (vs. V-JEPA 2 72.1%), DAVIS'17 71.6 J&F (close to DINOv3's 73.2; V-JEPA 2 collapses to 44.2).
- Streaming geometry: beats dedicated online 3D models on depth (Sintel Abs Rel 0.314 vs 0.341; BONN 0.072 vs 0.089; KITTI 0.136 vs 0.152) and camera pose (TUM Dynamics ATE 0.042 vs 0.051; ScanNet 0.128 vs 0.143).
- Spatial reasoning: VSI-Bench 70.6%, outperforming baselines with extra geometric encoders — 3D-RoPE alone suffices.
- Image tasks: minimal loss after adding temporal training (NYUv2 depth 0.089 vs DINOv3's 0.087; ADE20K 45.2 vs 46.1 mIoU).
- Short pretraining window (T=16); very long sequences (>1000 frames) unverified.
- Lightweight language decoder may trail dedicated multimodal LLMs on complex reasoning.
- Action-policy mapping remains a separate module for robot manipulation.
- 41 GB memory at T=512 still challenging for edge devices.
- Paper: https://arxiv.org/abs/2603.12265
Technical innovations
Causal Spatiotemporal Attention + Persistent KV-cache
| Metric | Bidirectional baseline | OmniStream | |---|---|---| | Latency | 0.998s | 0.067s (15× speedup) | | Memory | 15.14 GB | 7.26 GB | | T=128 / T=512 | OOM | 0.115s / 0.414s |
3D-RoPE (3D Rotary Position Embedding)
Multi-task Co-training
Results highlights
Why it matters for embodied AI
A single frozen backbone outputs semantic ("what"), spatial ("where"), and temporal ("how it moves") representations with real-time causal streaming and KV-cache memory of past frames — replacing the fragmented perception/SLAM/manipulation pipeline with one visual system deployable without fine-tuning.