Evidence-Backed Video Question Answering: Grounding Video LLM Answers in Spatio-Temporal Evidence

Forum topic · 小凯 · 2026-07-15

Summary

This paper introduces Evidence-Backed Video Question Answering (E-VQA), a new task requiring Video Large Language Models (Video LLMs) to output both a semantic answer and precise spatio-temporal evidence, including temporal segments and dense, tracked object segmentation masklets. The authors, from Salesforce Research, present ST-Evidence, the first human-verified benchmark covering both discriminative and generative pixel-level grounding in videos. Evaluations of state-of-the-art models reveal a critical decoupling between QA accuracy and genuine visual perception, showing that simply scaling model size cannot close the gap. To address this, they build an automated, scalable pipeline generating the 160k-sample ST-Evidence-Instruct dataset. Fine-tuning grounding Video LLMs on it yields substantial gains over the same-size UniPixel baseline (t-mean +27.2, J&F +13.8 on the 7B model). The work establishes a strong baseline for explainable, evidence-based video understanding.

Evidence-Backed Video Question Answering

Research area: Computer Vision (CV) Authors: Shijie Wang, Honglu Zhou, Ziyang Wang, Ran Xu, Caiming Xiong Published: 2026-07-13 arXiv: 2607.11862

Abstract

Current Video Large Language Models (Video LLMs) excel in question answering (QA) but largely operate as black boxes, providing textual answers without verifiable visual grounding. Existing explainability efforts rely on textual rationales or sparse bounding boxes, which struggle to capture complex video dynamics such as occlusions and non-rigid deformations. We propose Evidence-Backed Video Question Answering (E-VQA), a novel task requiring models to jointly output a semantic answer and precise spatio-temporal evidence: temporal segments and dense, tracked object segmentation masklets. To support this, we introduce ST-Evidence, the first human-verified benchmark for both discriminative and generative pixel-level grounding. Evaluations of state-of-the-art models reveal a critical decoupling between QA accuracy and true visual perception that scaling alone cannot bridge. We further develop a scalable automated generation pipeline to create the 160k-scale ST-Evidence-Instruct dataset. Fine-tuning a grounding Video LLM on it brings large improvements over the same-scale UniPixel baseline (t-mean +27.2 and J&F +13.8 on the 7B model), establishing a robust baseline for explainable, evidence-based video understanding.

Key Contributions

New task (E-VQA): models must jointly answer questions and provide verifiable spatio-temporal evidence — time segments plus dense tracked segmentation masklets.
ST-Evidence benchmark: the first human-verified benchmark for pixel-level grounding supporting both discriminative and generative settings.
Diagnostic finding: QA accuracy in SOTA Video LLMs is decoupled from genuine visual perception; scale alone does not close the gap.
ST-Evidence-Instruct dataset: 160k samples generated via a scalable automated pipeline.
Strong results: fine-tuned grounding Video LLM outperforms same-size UniPixel by t-mean +27.2 and J&F +13.8 (7B model), enabling handling of occlusions and non-rigid deformations that bounding-box approaches miss.