UniSem: Generalizable Semantic 3D Reconstruction from Sparse Unposed Images

Forum topic · 小凯 · 2026-03-19

Summary

UniSem is a unified feed-forward 3D Gaussian Splatting (3DGS) framework for semantic-aware 3D reconstruction from sparse, unposed images, presented in arXiv paper 2503.13837 by Guibiao Liao, Qian Ren, and Kaimin Liao (March 2025). It tackles two problems in prior work: over-complete Gaussian primitives that yield unstable geometry and poor depth under sparse-view supervision, and reliance on 2D segmenter features alone for semantic lifting, which produces incomplete 3D semantics in novel scenes. UniSem introduces Error-aware Gaussian Dropout (EGD), which suppresses redundancy-prone Gaussians using rendering error cues to stabilize geometry and improve depth estimation, and a Mixed Training Curriculum (MTC) that gradually blends 2D segmenter-lifted semantics with the model's emergent 3D semantic priors via object-level prototype alignment. Experiments on ScanNet and Replica show state-of-the-art depth prediction and open-vocabulary 3D segmentation across varying input view counts; with 16 input views, UniSem reduces depth relative error by 15.2% and improves open-vocabulary segmentation mAcc by 3.7% over strong baselines.

Overview

Research area: Computer Vision Authors: Guibiao Liao, Qian Ren, Kaimin Liao Published: 2025-03-18 arXiv: 2503.13837

Abstract

Semantic-aware 3D reconstruction from sparse, unposed images remains challenging for feed-forward 3D Gaussian Splatting (3DGS). Existing methods often predict an over-complete set of Gaussian primitives under sparse-view supervision, leading to unstable geometry and inferior depth quality. Meanwhile, they rely solely on 2D segmenter features for semantic lifting, which provides weak 3D-level and limited generalizable supervision, resulting in incomplete 3D semantics in novel scenes.

To address these issues, the authors propose UniSem, a unified framework that jointly improves depth accuracy and semantic generalization through two key components:

1. Error-aware Gaussian Dropout (EGD) — performs error-guided capacity control by suppressing redundancy-prone Gaussians using rendering error cues, producing geometrically stable Gaussian representations that improve depth estimation.

2. Mixed Training Curriculum (MTC) — progressively mixes 2D segmenter-lifted semantics with the model's own emergent 3D semantic priors, achieved through object-level prototype alignment, to enhance semantic consistency and completeness.

Results

Extensive experiments on ScanNet and Replica demonstrate that UniSem achieves superior performance in depth prediction and open-vocabulary 3D segmentation across varying numbers of input views. Notably, with 16-view input, UniSem:

Reduces depth relative error (Rel) by 15.2%
Improves open-vocabulary segmentation mAcc by 3.7%

outperforming strong baseline methods.

---

*Source: arXiv:2503.13837*