Paper Overview
Field: Computer Vision Authors: Guibiao Liao, Qian Ren, Kaimin Liao Published: 2025-03-18 arXiv: 2503.13837
Summary
Semantic-aware 3D reconstruction from sparse, unposed images remains challenging for feed-forward 3D Gaussian Splatting (3DGS). Existing methods often predict an over-complete set of Gaussian primitives under sparse-view supervision, leading to unstable geometry and inferior depth quality. Meanwhile, they rely solely on 2D segmenter features for semantic lifting, which provides weak 3D-level and limited generalizable supervision, resulting in incomplete 3D semantics in novel scenes.
To address these issues, UniSem is proposed as a unified framework that jointly improves depth accuracy and semantic generalization via two key components:
1. Error-aware Gaussian Dropout (EGD): performs error-guided capacity control by suppressing redundancy-prone Gaussians using rendering error cues, producing a geometrically stable Gaussian representation that improves depth estimation. 2. Mixed Training Curriculum (MTC): progressively mixes 2D segmenter-lifted semantics with the model's own emergent 3D semantic priors, achieved through object-level prototype alignment, to enhance semantic consistency and completeness.
Results
Extensive experiments on ScanNet and Replica demonstrate that UniSem achieves superior performance in depth prediction and open-vocabulary 3D segmentation across varying numbers of input views. Notably, with 16 input views, UniSem reduces depth Rel by 15.2% and improves open-vocabulary segmentation mAcc by 3.7%, outperforming strong baseline methods.
--- *Auto-collected on 2026-03-19*