OmniAssistBench: A Benchmark for Evaluating Omni-LLMs as Real-Time Video Assistants

Forum topic · 小凯 · 2026-08-25

Summary

OmniAssistBench is a new benchmark designed to evaluate omni-modal large language models (Omni-LLMs) acting as real-time, interactive video assistants. Unlike passive video understanding, interactive assistants must combine visual states, user goals, and prior knowledge to guide users toward specific goals. Because a model's unpredictable responses dynamically alter the user's subsequent actions, static offline datasets cannot capture this challenge. To handle diverging interaction paths, the benchmark provides models with predefined priors derived from source videos, requiring them to guide users along an identical route. Since genuine interactive videos are scarce, the dataset was constructed by reverse-engineering existing web videos: inferring logical user goals and segmenting videos into multi-turn episodes. The pipeline required over 1000 expert hours. Experiments show Gemini-3-Pro scores 66.4/100 and Qwen3-Omni-Instruct 51.2. Current models often understand user input but frequently give wrong or incomplete answers, struggle with visual cues like gestures, lose multi-turn context, and fail to delay responses before goal-relevant events. (arXiv: 2608.21360)

Overview

Research area: Computer Vision Authors: Xianyun Sun, Chaoyou Fu, Zhengye Zhang, Feiyang Duan, Qingyuan Cao, Yonghui Niu, Sihang Yuan, Ge Zhang, Caifeng Shan Published: 2026-08-21 arXiv: 2608.21360

Summary

Recent omni-modal large language models (Omni-LLMs) show great potential as real-time video assistants that continuously perceive environments and guide users toward specific goals. Unlike traditional passive video understanding, interactive assistants should actively combine visual states, user goals, and prior knowledge to provide effective help.

Evaluating this capability is challenging: a model's unpredictable responses dynamically change the user's subsequent actions, which static offline datasets cannot accommodate. To address this bottleneck, the authors introduce OmniAssistBench.

Key design choices

Handling diverging interaction paths: Since the same user goal can be achieved through many routes, models are provided with predefined priors derived from the source video and must guide the user along the exact same route.
Data construction via reverse engineering: Because real interactive videos are scarce, the dataset is built by reverse-engineering existing web videos — inferring logical user goals and segmenting videos into multi-turn episodes to simulate continuous interaction.
Rigor: The pipeline required over 1000 expert hours.

Results

| Model | Score (out of 100) | |---|---| | Gemini-3-Pro (proprietary) | 66.4 | | Qwen3-Omni-Instruct (open-source) | 51.2 |

Although current models can generally understand user input, they frequently produce wrong or incomplete answers. Specifically, they:

Struggle with visual prompts such as hand gestures;
Fail to maintain historical context across multi-turn interactions;
Cannot delay responses appropriately before goal-relevant events.

These findings indicate substantial room for improvement before models can become reliable real-time assistants.

--- *Auto-collected on 2026-08-25*

OmniAssistBench: A Benchmark for Evaluating Omni-LLMs as Real-Time Video Assistants

Summary

Overview

Summary

Key design choices

Results

Tags