Paper Overview
Research Area: 3D Vision Authors: Jameel Hassan, Yasiru Ranasinghe, Vishal Patel arXiv: 2507.00004
Summary
3D Gaussian Splatting (3DGS) has emerged at the forefront of 3D scene reconstruction. Extending 3DGS with language-driven, open-vocabulary understanding has gained significant attention for real-world applications such as embodied AI.
Recent methods achieve this by learning an instance feature attribute and assigning semantics by distilling high-dimensional CLIP features directly into the scene representation. However, the instance grouping mechanisms of these methods either require a predefined number of instances or suffer from noise in their bottom-up grouping strategies. Furthermore, the reliance on CLIP restricts semantic understanding to simple noun phrases, preventing complex spatial reasoning and referential expression grounding.
GaussDet
This paper presents GaussDet, a method that circumvents the need for dense CLIP features by leveraging a discrete open-vocabulary 2D object detector with referring capabilities.
Key ideas:
- Instance features are learned for individual Gaussians, decomposing the scene into 3D instance groups.
- These groups are rendered, and multi-view 2D detections are aggregated via semantic voting to produce a robust View-Aggregated Semantic Distribution (VASD) for each 3D instance.
- The view-aggregation strategy serves as a strong regularizer, mitigating pseudo-labels caused by low-quality instance grouping.
Results
Extensive evaluations on two key tasks — open-vocabulary segmentation (LeRF-OVS, ScanNet) and referring expression grounding (Ref-LeRF) — demonstrate consistent improvements over existing methods. Notably, GaussDet achieves a significant 16.7% mIoU improvement in a strict zero-shot setting.
--- *Auto-collected on 2026-07-01*