CiteVQA: A Benchmark That Forces AI to Cite Real Evidence in Documents

Forum topic · 小凯 · 2026-05-25

Summary

CiteVQA, released in May 2026 (arXiv:2605.12882), is a benchmark for evaluating evidence attribution in multimodal large language models (MLLMs) on document understanding tasks. Unlike traditional evaluations that only check answer correctness, CiteVQA requires models to pinpoint the exact bounding-box coordinates of the evidence supporting their answers. It covers 1,897 questions across seven professional domains, with all answers and coordinates verified through both machine ablation experiments and human expert cross-checks. Its Strict Attributed Accuracy (SAA) metric awards a score only when the predicted answer is correct and the Intersection over Union (IoU) between predicted and ground-truth evidence regions exceeds a strict threshold. The benchmark exposes significant weaknesses: the best closed-source model, Gemini-3.1-Pro, achieves only 76 points, while leading open-source models score a mere 22.5. This reveals that current models often rely on parametric "guessing" rather than genuine grounding — a problem known as attribution hallucination, which is especially dangerous in legal, financial, and medical settings where auditable evidence matters.

When AI Answers Right but Cites Wrong

Asking AI to read documents and answer questions is routine now — but there's a hidden flaw. Models often produce correct answers while pointing to evidence pages that say nothing relevant. In fields like law, finance, and healthcare, this "right answer, no evidence" behavior is a ticking time bomb.

The root cause lies in how we've traditionally evaluated these models: only checking the final answer, never the reasoning or cited passages. This lets models win by powerful "guessing" — a phenomenon researchers call Attribution Hallucination.

The CiteVQA Benchmark (May 2026)

CiteVQA imposes a strict rule: answers alone aren't enough — models must mark the bounding-box coordinates of the supporting evidence in the document.

Scale: 1,897 challenging questions across seven professional domains.
Strict scoring (SAA): Evidence pointing wrong by even a bit means zero points, regardless of answer quality.
Expert verification: All ground-truth answers and coordinates were cross-validated via machine ablation experiments and human expert review.

The core evaluation logic:

\text{SAA} = \mathcal{I}(A_{\text{pred}} = A_{gt}) \wedge \text{IoU}(C_{\text{pred}}, C_{gt}) > \tau

A score is granted only when the answer matches ground truth and the IoU of predicted vs. ground-truth evidence coordinates exceeds a strict threshold.

Results: A Wake-Up Call

Top closed-source model Gemini-3.1-Pro reached only 76 points. Leading open-source models scored a mere 22.5 — showing that genuine evidence-grounding capability remains far behind, and past lenient benchmarks had "spoiled" our models.

With CiteVQA, AI must finally abandon groundless claims: every conclusion must be backed by real, locatable evidence. One piece of evidence, one piece of statement.

Reference

Paper: CiteVQA: Benchmarking Evidence Attribution for Trustworthy Document Intelligence
Released: May 18, 2026
arXiv: arXiv:2605.12882
Core contribution: Solves "answer-without-evidence" / false attribution in MLLM document understanding via meta-level element coordinate citation tasks and the Strict Attributed Accuracy (SAA) metric, establishing an auditable document intelligence benchmark.