UniD: Unified Video Dense Prediction from Disjoint Data

Forum topic · 小凯 · 2026-07-26

Summary

UniD is a unified video model that jointly predicts eight dense scene properties—depth, surface normals, semantic segmentation, boundaries, human parts, albedo, shading, and materials—from disjoint, domain-specific datasets. Existing unified scene understanding systems are limited by fragmented task-specific annotations: they either restrict training to fully co-annotated data or pay the heavy computational cost of pseudo-labeling. UniD avoids both by using a simple distillation step in which per-task experts supervise a unified backbone through lightweight task projectors, eliminating the need for annotation overlap or pseudo-labels. The key insight is that strong visual priors from a pretrained diffusion model are sufficient to bridge the domain gap introduced by disjoint training sources, enabling robust generalization to scene-task combinations never seen during training. UniD achieves competitive performance against both per-task experts and multi-task baselines, generalizes strongly to out-of-distribution scenes, and improves temporal consistency and cross-task consistency. Paper: arXiv 2507.20486; code and video results at https://unid-video.github.io/.

Overview

Field: Computer Vision
Authors: Yihong Sun, Seoung Wug Oh, Jiahui Huang
arXiv: 2507.20486
Project page: https://unid-video.github.io/

Key Points

Scene understanding requires simultaneous prediction of geometry, appearance, and semantics, but task-specific annotations are fragmented across incompatible, domain-specific datasets.
Current unified systems work around this by training only on fully co-annotated data or by incurring the large computational cost of pseudo-labeling.
The authors introduce UniD, a unified video model that jointly predicts eight dense scene properties—depth, surface normals, semantic segmentation, boundaries, human parts, albedo, shading, and materials—all learned from disjoint, domain-specific datasets.
A simple yet effective distillation step lets per-task experts supervise a unified backbone through lightweight task projectors, eliminating the need for annotation overlap or pseudo-labels.
Core insight: the strong visual priors of a pretrained diffusion model are sufficient to bridge the domain gap introduced by disjoint training sources, enabling robust generalization to scene-task combinations never seen during training.
UniD achieves competitive performance against both per-task experts and multi-task baselines, with strong out-of-distribution generalization and improved temporal and cross-task consistency.

Abstract (Original)

Scene understanding requires simultaneous prediction about geometry, appearance, and semantics. However, existing task-specific annotations are fragmented across incompatible, domain-specific datasets. Current unified systems circumvent this by restricting training to fully co-annotated data, or by incurring the large computational cost of pseudo-labeling. To mitigate this, we introduce UniD, a unified video model that jointly predicts eight dense scene properties—depth, surface normals, semantic segmentation, boundaries, human parts, albedo, shading, and materials—all learned from disjoint, domain-specific datasets. We propose a simple yet effective distillation step in which per-task experts supervise a unified backbone through lightweight task projectors, eliminating the need for annotation overlap or pseudo-labels.