SR-ReaL: Dual-Path Reasoning Reinforcement for Spatial Vision-Language Models

Forum topic · 小凯 · 2026-06-19

Summary

SR-ReaL is a spatial vision-language model framework from HKU, NVIDIA, and UCSD that is the first to support two complementary reasoning paths in a single model: Language-Only Reasoning (LOR) for step-by-step textual deduction of spatial relations, and Detect-Then-Reason (DTR), which first extracts 3D coordinates via region tokens before performing quantitative geometric computation. Trained with a two-stage pipeline—cold-start supervised fine-tuning on roughly 1 million samples (including ~40k chain-of-thought data mixed with grounding and general multimodal QA) followed by DAPO-style GRPO reinforcement learning with format, accuracy, and 3D detection rewards—the model achieves 61.9 on SPAR-Bench versus a 33.4 baseline (+28.5). A key finding is mutual reinforcement: jointly training both paths outperforms training either alone, with DTR data improving LOR by strengthening spatial representations and LOR data improving DTR by preserving qualitative spatial intuition. Ablations confirm that the region-to-3D interface and detection reward nearly halve 3D localization error. Out-of-distribution tests show direct (non-CoT) inference remains strong, indicating preserved general perception.

SR-ReaL: Reinforcing Dual-Path Reasoning in Spatial Vision-Language Models

> Paper: Reinforcing Dual-Path Reasoning in Spatial Vision Language Models > Authors: Yatai Ji, An-Chieh Cheng, Yang Fu, et al. (HKU, NVIDIA, UCSD) > Link: https://arxiv.org/abs/2606.17539 > Core contribution: First spatial VLM to simultaneously support Language-Only Reasoning (LOR) and Detect-Then-Reason (DTR) paths, mutually enhanced within a unified reinforcement learning framework.

The Problem: One Question, Two Strategies

Spatial VLMs have improved rapidly but still struggle with complex spatial reasoning—multi-step questions like:

"Walking from the living room to the bedroom, how many doors do you pass?"
"How far is this chair from the table? What obstacles are in between?"
"From this viewpoint, is object A to the left or right of object B?"

The authors observe two key phenomena:

1. Some problems are solvable with pure language reasoning (e.g., "A is left of B, B is right of C—where is A relative to C?") via step-by-step linguistic deduction. 2. Some problems require explicit 3D geometry (e.g., "How many meters apart are these objects?")—pure language reasoning can only guess.

Gap in prior work: existing frameworks support only one strategy; none support both, let alone make them reinforce each other.

Dual-Path Design

LOR (Language-Only Reasoning): step-by-step textual deduction, suited to relational inference, viewpoint reasoning, and spatial imagination.
DTR (Detect-Then-Reason): first extract 3D coordinates (center point or bounding box) via region tokens, then perform quantitative reasoning—suited to depth estimation, distance inference, and precise localization.

The model routes between paths based on prompt instructions.

Region-to-3D Grounding

Directly predicting 3D coordinates from text is extremely hard. SR-ReaL bridges semantics and geometry with region tokens:

1. Input: a text description (e.g., "the red chair") is mapped to a visual region (mask/bounding box), producing a region token. 2. 3D prediction: region tokens are decoded into 3D center (x, y, z) or 3D bounding boxes (center + size + orientation). 3. Reasoning: explicit coordinates enable geometric computation.

This decouples semantic parsing from 3D perception. Ablations show removing the region-to-3D interface drops DTR performance from 60.6 to 59.3 and nearly doubles 3D localization error (0.45 → 0.67).

Two-Stage Training

Stage 1: Cold-Start SFT (~1M samples)

| Data | Amount | Purpose | |------|--------|---------| | CoT-LOR | 30k | chain-of-thought linguistic reasoning | | CoT-DTR | 10k | detect-then-reason with 3D cues | | Complex spatial tasks | 20k | navigation, interaction, layout reasoning | | 2D Grounding | RefCOCO | text-to-2D-box prediction | | 3D Grounding | Omni3D/OmniNOCS | region-to-3D mapping | | Region-prompted QA | SRGPT | local spatial understanding | | General multimodal QA | LLaVA-1.5 | preserve general ability |

Key finding: training on CoT data alone causes rapid degradation of general multimodal ability; mixing general data is a prerequisite for stable RL.

Stage 2: Reinforcement Learning (DAPO-style GRPO)

Rewards include:

Format: <think>...</think><answer>...</answer> for LOR; <detect>...</detect> prepended for DTR.
Accuracy: exact scoring for multiple choice; exponentially smoothed relative error for free-form answers.
3D detection (DTR only): discretized reward based on distance between predicted and ground-truth 3D centers.

Online filtering (from DAPO): remove rollout groups where all samples receive identical total rewards, then resample.

Results

SPAR-Bench (20 subtasks)

| Model | SPAR-Bench avg | EmbSpatial | SAT | |-------|---------------|-----------|-----| | SR-3D (baseline) | 33.4 | 72.5 | 63.0 | | Ours-LOR | 60.5 | 79.2 | 68.7 | | Ours-DTR | 61.9 | 81.3 | - |

DTR improves over baseline by +28.5—nearly doubling performance.

Mutual Reinforcement

| Training | SPAR (LOR) | SPAR (DTR) | EmbSpatial (LOR) | EmbSpatial (DTR) | |----------|-----------|-----------|------------------|------------------| | LOR only | 58.0 | - | 75.9 | - | | DTR only | - | 57.2 | - | 71.4 | | Joint | 58.7 | 60.8 | 77.6 | 78.8 |

DTR data improves LOR: explicit geometric representations strengthen underlying spatial features.
LOR data improves DTR: it restores the "spatial intuition" lost when over-relying on quantitative computation.
Joint training wins on both paths.

Ablations

| DTR config | SPAR | EmbSpatial | 3D loc. error | |-----------|------|-----------|---------------| | w/o detection reward | 59.9 | 76.0 | 0.78 | | w/o region-to-3D | 59.3 | 74.8 | 0.67 | | Full | 60.6 | 78.5 | 0.45 |

Cold-start vs RL: cold-start alone teaches CoT format but generalizes weakly; RL-only scores well but produces inconsistent chains of thought; the full pipeline achieves the best performance and consistency.

Out-of-Distribution (BLINK, RealWorldQA, CVBench)

Interestingly, direct inference (no CoT) beats CoT on OOD benchmarks (87.4 vs 80.4 on BLINK-Spatial), suggesting training enhanced complex reasoning without damaging the model's original spatial perception—flexible CoT usage remains possible.

Highlights

1. Unified dual-path design: one model, one checkpoint, prompt-guided switching between LOR and DTR. 2. RL shapes reasoning behavior, not just answers: both paths regularize each other in a single RL framework. 3. Data quality > quantity: only ~40k CoT samples (with two-stage filtering for answer match and logical consistency) sufficed for stable RL. 4. Complete perception-to-reasoning chain: image + question → semantic parsing (region token) → 3D perception (coordinates) → geometric reasoning → answer.

Limitations and Future Work

Mainly validated in single-view settings; multi-view reasoning has room to grow.
Primarily indoor scenes (SPAR, EmbSpatial); limited outdoor data (NuScenes).
Absolute accuracy on some subtasks (e.g., viewpoint change) still below 50%.
Training requires 32× A100 GPUs.

Future directions: video input and temporal reasoning, more reasoning-path combinations beyond the LOR/DTR binary, and integration with world models for spatial planning.

Conclusion

SR-ReaL represents a paradigm shift—from single-strategy to multi-path complementary spatial reasoning, applicable beyond vision to mathematics, science, and everyday reasoning. As the paper states:

> "Jointly training both reasoning modes fosters mutual reinforcement, with each mode benefiting from the other's supervision."

References

Ji et al., "Reinforcing Dual-Path Reasoning in Spatial Vision Language Models", arXiv:2606.17539, 2026
Cheng et al., "SpatialRGPT", 2024
Cheng et al., "SR-3D", 2025
DeepSeek-AI, "DeepSeek-R1", 2025
Zhang et al., "SPAR", 2025