GaussDet: Open-Vocabulary and Referring Segmentation for 3D Gaussians Using 2D Detectors

Forum topic · 小凯 · 2026-07-01

Summary

GaussDet is a new method for adding language-driven, open-vocabulary understanding to 3D Gaussian Splatting (3DGS) scene representations, presented in arXiv paper 2507.00004 by researchers from Johns Hopkins (Jameel Hassan, Yasiru Ranasinghe, Vishal Patel). Instead of distilling dense CLIP features directly into 3D scenes, GaussDet leverages a discrete open-vocabulary 2D object detector with referring capability. The model learns instance features for individual Gaussians to decompose scenes into 3D instance groups, renders these groups, and aggregates multi-view 2D detections into a robust View-Aggregated Semantic Distribution (VASD) for each 3D instance. This aggregation acts as a regularizer, suppressing noisy pseudo-labels from imperfect instance grouping. The approach supports zero-shot extension from simple noun-phrase queries to complex referring expression grounding, overcoming CLIP's limits on spatial reasoning. Evaluations on open-vocabulary segmentation (LeRF-OVS, ScanNet) and referring grounding (Ref-LeRF) show consistent improvements over prior methods, including a notable 16.7% mIoU gain in strict zero-shot settings.

Paper Overview

Field: 3D Vision
Authors: Jameel Hassan, Yasiru Ranasinghe, Vishal Patel
arXiv: 2507.00004

Background

3D Gaussian Splatting (3DGS) has become a leading technique for 3D scene reconstruction. Extending 3DGS with language-driven, open-vocabulary understanding has drawn significant attention for real-world applications such as embodied AI.

Recent methods achieve semantic assignment by learning an instance feature attribute and distilling high-dimensional CLIP features directly into the scene representation. However:

Their instance grouping mechanisms either require a predefined number of instances or suffer from noise in bottom-up grouping strategies.
Reliance on CLIP restricts semantic understanding to simple noun phrases, preventing complex spatial reasoning and referring expression grounding.

Method: GaussDet

GaussDet avoids the need for dense CLIP features by exploiting a discrete, open-vocabulary 2D object detector with referring capabilities:

1. Instance features are learned for individual Gaussians, decomposing the scene into 3D instance groups. 2. These groups are rendered, and semantics from multi-view 2D detections are aggregated via voting into a robust View-Aggregated Semantic Distribution (VASD) for each 3D instance. 3. The view-aggregation strategy acts as a strong regularizer, mitigating pseudo-label noise caused by low-quality instance grouping.

The method enables direct zero-shot extension from simple language queries to complex referring expression grounding.

Results

Evaluations on two key tasks demonstrate consistent improvements over existing methods:

Open-vocabulary segmentation: LeRF-OVS, ScanNet
Referring expression grounding: Ref-LeRF

Most notably, GaussDet achieves a significant 16.7% mIoU improvement in the strict zero-shot setting.

---

*Auto-collected on 2026-07-01*