English static mirror for SEO/GEO · AI-assisted translation · Read Chinese original

OmniStream Explained: A Unified Visual Foundation Model for Continuous Video Streams

Forum topic · 小凯 · 2026-05-11

Summary

OmniStream is a unified vision foundation model from Shanghai Jiao Tong University and Oxford VGG designed to handle semantics, spatial geometry, and temporal reasoning in continuous video streams. Unlike fragmented vision models specialized in single tasks, OmniStream aims to be deployed frozen across 29 diverse tasks without fine-tuning. Its key innovations include causal spatiotemporal attention with a persistent KV-cache, enabling strict per-frame online processing (0.067s latency at T=64 versus 0.998s for bidirectional recomputation, a 15x speedup with halved memory), and a 3D Rotary Position Embedding (3D-RoPE) that extends position encoding to height, width, and time, enabling zero-shot length extrapolation from a 16-frame training window to 110 frames. Trained jointly on 29 datasets spanning representation learning, streaming geometric reconstruction, and vision-language alignment, OmniStream matches or beats specialized models: 68.5% on SSv2, 71.6 J&F on DAVIS'17, better depth estimation than dedicated online 3D models on Sintel and KITTI, and 70.6% on VSI-Bench without extra geometric encoders. The model offers embodied agents a single visual backbone delivering semantic, spatial, and temporal representations with real-time streaming capability.

OmniStream: Mastering Perception, Reconstruction and Action in Continuous Streams

> Research date: 2025-05-11 | arXiv: https://arxiv.org/abs/2603.12265 > Authors: Yibin Yan, Jilan Xu, Shangzhe Di, Haoning Wu, Weidi Xie > Institutions: School of AI, Shanghai Jiao Tong University; Shanghai Innovation Institute; Oxford VGG

Key points

  • Motivation: Current vision foundation models are fragmented — specialized in image semantics (DINOv3), offline temporal modeling (V-JEPA 2), or spatial geometry (CUT3R). No single model handles semantics + space + time together, which is critical for real-time embodied agents.
  • Core claim: A single visual backbone, kept strictly frozen, can be deployed across 29 different tasks without any fine-tuning.
  • Technical innovations

    Causal Spatiotemporal Attention + Persistent KV-cache

  • Frame *t* only attends to frames ≤ t; per-frame online processing with cached keys/values.
  • Benchmarks (T=64, 224×224, single H800):
  • | Metric | Bidirectional baseline | OmniStream | |---|---|---| | Latency | 0.998s | 0.067s (15× speedup) | | Memory | 15.14 GB | 7.26 GB | | T=128 / T=512 | OOM | 0.115s / 0.414s |

    3D-RoPE (3D Rotary Position Embedding)

  • Extends RoPE to height, width, and time axes.
  • Enables zero-shot length extrapolation: trained on T=16 frames, infers on 110 frames (7× extrapolation).
  • Multi-task Co-training

  • Joint pretraining on 29 datasets covering:
  • Static + temporal representation learning (ImageNet, K400, SSv2, DAVIS'17)
  • Streaming geometric reconstruction (Sintel, BONN, KITTI, TUM, ScanNet)
  • Vision-language alignment (VSI-Bench, OVO-Bench)
  • Backbone initialized from DINOv3; dual DPT heads predict depth/ray maps and camera pose; lightweight autoregressive language decoder links visual tokens to language.
  • Results highlights

  • Video understanding: SSv2 68.5% (vs. DINOv3 54.0%), K400 74.2% (vs. V-JEPA 2 72.1%), DAVIS'17 71.6 J&F (close to DINOv3's 73.2; V-JEPA 2 collapses to 44.2).
  • Streaming geometry: beats dedicated online 3D models on depth (Sintel Abs Rel 0.314 vs 0.341; BONN 0.072 vs 0.089; KITTI 0.136 vs 0.152) and camera pose (TUM Dynamics ATE 0.042 vs 0.051; ScanNet 0.128 vs 0.143).
  • Spatial reasoning: VSI-Bench 70.6%, outperforming baselines with extra geometric encoders — 3D-RoPE alone suffices.
  • Image tasks: minimal loss after adding temporal training (NYUv2 depth 0.089 vs DINOv3's 0.087; ADE20K 45.2 vs 46.1 mIoU).
  • Why it matters for embodied AI

    A single frozen backbone outputs semantic ("what"), spatial ("where"), and temporal ("how it moves") representations with real-time causal streaming and KV-cache memory of past frames — replacing the fragmented perception/SLAM/manipulation pipeline with one visual system deployable without fine-tuning.

    Limitations

  • Short pretraining window (T=16); very long sequences (>1000 frames) unverified.
  • Lightweight language decoder may trail dedicated multimodal LLMs on complex reasoning.
  • Action-policy mapping remains a separate module for robot manipulation.
  • 41 GB memory at T=512 still challenging for edge devices.
  • Reference

  • Paper: https://arxiv.org/abs/2603.12265

Tags

#omnistream#vision-foundation-model#causal-attention#kv-cache#3d-rope#video-streaming#embodied-ai#depth-estimation

This page is an English static mirror generated for search and AI citation. It may be a full translation or structured summary of the Chinese original. Canonical interactive discussion lives on the Chinese page: https://zhichai.net/topic/177619809