FastBench: Benchmarking High-Dynamic Perception in Streaming Video LLMs

Forum topic · 小凯 · 2026-10-11

Summary

FastBench is a new benchmark for evaluating high-dynamic perception in streaming Video Large Language Models (VLMs). Existing benchmarks focus on low-dynamic scenarios, while sparse sampling at 1-2 FPS misses fast events under bounded context budgets. FastBench uses a trajectory-grounded pipeline: QA generation from high-FPS clips, filtering of questions answerable at 2 FPS, answer verification with SAM3 and CoTracker3 trajectories, and three rounds of human inspection. It contains 306 QA pairs spanning eight domains, six capabilities, and forward/instant/backward temporal scopes with human-annotated evidence intervals. The paper also introduces ProactiveFrame, a training-free baseline that adjusts input frame rates via text tokens, using a two-level sliding window that keeps recent observations at high FPS while downsampling older history. Experiments show major limitations: the strongest model, Gemini-3.5-Flash, scores only 50.7%; denser sampling lifts Qwen3-VL-8B from 32.9% (2 FPS) to 44.6% (24 FPS), with saturating gains as history is compressed; ProactiveFrame outperforms sparse uniform sampling by 5.4 and 1.5 points but remains far below oracle-guided focusing. Code and data are available at https://github.com/Ashone3/FastBench.

Overview

Field: NLP Authors: Yuxuan Hu, Weikang Shi, Yang Bo, Xudong Lu, Xintong Guo, Shuhan Li, Yuyang He, Huankang Guan, Peiwen Sun, Yunqiao Yang, Wenbo Li, Rui Liu, Hongsheng Li Published: 2026-10-08 arXiv: 2610.12427

Key Points

Streaming Video Large Language Models (VLMs) enable continuous video understanding, yet existing benchmarks focus on low-dynamic scenarios. Under bounded context budgets, models must balance temporal history, spatial resolution, and temporal granularity; sparse sampling at 1--2 FPS misses fast events.

FastBench evaluates high-dynamic perception in real-world video streams.
Its trajectory-grounded pipeline combines: QA generation from high-FPS clips, filtering of questions answerable at 2 FPS, answer verification using SAM3 and CoTracker3 trajectories, and three rounds of human inspection.
The benchmark contains 306 QA pairs across eight domains, six capabilities, and forward, instant, and backward temporal scopes, with human-annotated evidence intervals.
ProactiveFrame is proposed as a training-free baseline that adjusts input frame rates via text tokens; a two-level sliding window preserves recent high-FPS observations while downsampling earlier ones into sparse history.

Findings

The strongest model, Gemini-3.5-Flash, achieves only 50.7%.
Denser sampling raises Qwen3-VL-8B from 32.9% at 2 FPS to 44.6% at 24 FPS, but gains saturate as history is compressed.
ProactiveFrame outperforms sparse uniform sampling by 5.4 and 1.5 percentage points, but remains well below oracle-guided focusing, indicating current VLMs struggle to decide from the stream alone when finer temporal perception is needed.

FastBench provides a testbed for high-dynamic streaming video understanding.

Code and data: https://github.com/Ashone3/FastBench