Beyond Referring Expressions: Scenario-Based Visual Grounding with the RSC Benchmark and ScenGround

Forum topic · 小凯 · 2026-04-05

Summary

A forum post introduces an arXiv paper (2604.02323) proposing scenario-based visual grounding, a setting more challenging than standard referring expression comprehension. Instead of matching explicitly named object categories, models must infer targets from object roles, user intentions, and relational context. The authors present Referring Scenario Comprehension (RSC), a benchmark with paragraph-length queries describing roles, goals, and contextual cues, including deliberate distractor references. Every instance carries interpretable difficulty tags covering uniqueness, clutter, size, overlap, and position for fine-grained failure analysis. RSC includes roughly 31k training examples, 4k in-domain test examples, and a 3k out-of-distribution split with unseen categories. The paper also introduces ScenGround, a curriculum reasoning baseline combining supervised warm-starting with difficulty-aware reinforcement learning. Experiments show scenario-based queries expose systematic failures hidden by standard benchmarks, and curriculum training improves performance on hard slices while transferring to standard benchmarks.

Paper Overview

Field: Computer Vision Authors: Ruozhen He, Nisarg A. Shah, Qihua Dong Published: 2026-04-02 arXiv: 2604.02323

Abstract

Existing visual grounding benchmarks primarily evaluate alignment between image regions and literal referring expressions, where models can often succeed by matching a prominent named category. We explore a complementary and more challenging setting of scenario-based visual grounding, where the target must be inferred from roles, intentions, and relational context rather than explicit naming.

We introduce Referring Scenario Comprehension (RSC), a benchmark designed for this setting. The queries in this benchmark are paragraph-length texts that describe object roles, user goals, and contextual cues, including deliberate references to distractor objects that often require deep understanding to resolve. Each instance is annotated with interpretable difficulty tags for uniqueness, clutter, size, overlap, and position, which expose distinct failure modes and support fine-grained analysis.

RSC contains approximately 31k training examples, 4k in-domain test examples, and a 3k out-of-distribution split with unseen object categories.

ScenGround

The authors further propose ScenGround, a curriculum reasoning method serving as a reference point for this setting, combining:

Supervised warm-starting
Difficulty-aware reinforcement learning

Findings

Experiments show that scenario-based queries expose systematic failures in current models that standard benchmarks do not reveal, and that curriculum training improves performance on challenging slices and transfers to standard benchmarks.

---

*Auto-collected on 2026-04-05.*