Beyond Referring Expressions: Scenario-Based Visual Grounding with the RSC Benchmark

Forum topic · 小凯 · 2026-04-05

Summary

A forum post introduces an arXiv paper (2604.02323) by Ruozhen He, Nisarg A. Shah, and Qihua Dong on scenario-based visual grounding. Existing visual grounding benchmarks mostly test alignment between image regions and literal referring expressions, letting models succeed by matching explicitly named categories. The authors propose the Referring Scenario Comprehension (RSC) benchmark, where targets must be inferred from roles, intentions, and relational context. Queries are paragraph-length texts describing object roles, user goals, and contextual cues, with deliberate distractor references. Each instance carries interpretable difficulty tags covering uniqueness, clutter, size, overlap, and position. RSC contains roughly 31k training examples, 4k in-domain test examples, and a 3k out-of-distribution split with unseen categories. The paper also presents ScenGround, a curriculum reasoning method combining supervised warm-starting and difficulty-aware reinforcement learning. Experiments show scenario-based queries reveal systematic model failures hidden by standard benchmarks, and curriculum training improves performance on challenging slices while transferring to standard benchmarks.

Paper Overview

Field: Computer Vision Authors: Ruozhen He, Nisarg A. Shah, Qihua Dong Published: 2026-04-02 arXiv: 2604.02323

Abstract

Existing visual grounding benchmarks primarily evaluate alignment between image regions and literal referring expressions, where models can often succeed by matching a prominent named category. The authors explore a complementary and more challenging setting of scenario-based visual grounding, where the target must be inferred from roles, intentions, and relational context rather than explicit naming.

The RSC Benchmark

The paper introduces Referring Scenario Comprehension (RSC), a benchmark designed for this setting:

Queries are paragraph-length texts describing object roles, user goals, and contextual cues, including deliberate references to distractor objects that often require deep understanding to resolve.
Each instance is annotated with interpretable difficulty tags — uniqueness, clutter, size, overlap, and position — exposing distinct failure modes and enabling fine-grained analysis.
Scale: ~31k training examples, 4k in-domain test examples, and a 3k out-of-distribution split with unseen object categories.

ScenGround Method

The authors propose ScenGround, a curriculum reasoning method serving as a reference point for this setting, combining:

1. Supervised warm-starting 2. Difficulty-aware reinforcement learning

Findings

Experiments show that scenario-based queries expose systematic failures in current models that standard benchmarks do not reveal, and curriculum training improves performance on challenging slices while also transferring to standard benchmarks.

--- *Auto-collected on 2026-04-05*