Paper Overview
- Research Area: Computer Vision (CV)
- Authors: Xudong Lu, Yang Bo, Jinpeng Chen
- Unified streaming architecture: continuous video processing without decoupled trigger-response pipelines
- Real-time QA + proactive responses: supports open-ended question answering and long-horizon interaction
- Full-stack design: context management, data construction, training objectives, and deployment optimization
- Practical deployment: real-time demo with ASR/TTS at 2 FPS on two 80GB accelerators
- Open release: model and real-time inference framework publicly available
Abstract
Video Large Language Models (VideoLLMs) have achieved strong performance on many video understanding tasks, but most existing systems remain offline and are not well-suited for live video streams that require continuous observation and timely response. Recent streaming VideoLLMs have made progress, yet current approaches often rely on decoupled trigger-response pipelines or are limited to captioning-style narration, reducing their effectiveness for open-ended question answering and long-horizon interaction.We propose AURA (Always-On Understanding and Real-Time Assistance), an end-to-end streaming visual interaction framework that enables a unified VideoLLM to continuously process video streams and support both real-time question answering and proactive responses. AURA integrates context management, data construction, training objectives, and deployment optimization for stable long-horizon streaming interaction. It achieves state-of-the-art performance on streaming benchmarks and supports a real-time demo system with ASR and TTS running at 2 FPS on two 80G accelerators. We release the AURA model together with a real-time inference framework to facilitate future research.