Beyond Referring Expressions: Scenario Comprehension Visual Grounding (RSC Benchmark & ScenGround)

Forum topic · 小凯 · 2026-04-04

Summary

A 2025 arXiv paper (2504.01259) by Ruozhen He, Nisarg A. Shah, and Qihua Dong introduces Referring Scenario Comprehension (RSC), a benchmark for scenario-based visual grounding where targets must be inferred from roles, intentions, and relational context rather than explicit naming. RSC queries are paragraph-length texts describing object roles, user goals, and contextual cues, including deliberate distractor references. Each instance carries interpretable difficulty tags (uniqueness, clutter, size, overlap, position) exposing distinct failure modes and enabling fine-grained analysis. The benchmark contains roughly 31k training samples, 4k in-domain test samples, and a 3k out-of-distribution split with unseen object categories. The authors also propose ScenGround, a curriculum-based reasoning method combining supervised warm-up with difficulty-aware reinforcement learning. Experiments show scenario-based queries reveal systematic failures that standard benchmarks miss, and curriculum training improves performance on challenging slices while transferring to standard benchmarks.

Paper Overview

Field: Computer Vision
Authors: Ruozhen He, Nisarg A. Shah, Qihua Dong
Published: 2025-04-01
arXiv: 2504.01259

Abstract

Existing visual grounding benchmarks primarily evaluate alignment between image regions and literal referring expressions, where models can often succeed by matching a prominent named category. We explore a complementary and more challenging setting of scenario-based visual grounding, where the target must be inferred from roles, intentions, and relational context rather than explicit naming.

We introduce Referring Scenario Comprehension (RSC), a benchmark designed for this setting. The queries in this benchmark are paragraph-length texts that describe object roles, user goals, and contextual cues, including deliberate references to distractor objects that often require deep understanding to resolve. Each instance is annotated with interpretable difficulty tags for uniqueness, clutter, size, overlap, and position, which expose distinct failure modes and support fine-grained analysis.

RSC contains approximately 31k training samples, 4k in-domain test samples, and a 3k out-of-distribution split covering unseen object categories. We further propose ScenGround, a curriculum-based reasoning method serving as a reference point for this setting, combining supervised warm-up with difficulty-aware reinforcement learning.

Experiments demonstrate that scenario-based queries expose systematic failures in current models that standard benchmarks cannot reveal, and that curriculum training improves performance on challenging slices while transferring to standard benchmarks.

Key Contributions

RSC benchmark: paragraph-length scenario queries requiring inference over roles, intentions, and relational context, with distractor objects deliberately included.
Difficulty tags: interpretable per-instance annotations (uniqueness, clutter, size, overlap, position) enabling fine-grained failure analysis.
Scale: ~31k training, 4k in-domain test, 3k OOD (unseen categories) samples.
ScenGround: supervised warm-up + difficulty-aware RL curriculum, improving challenging-slice performance and transferring to standard grounding benchmarks.

---

*Auto-collected on 2026-04-04*