UniD: Unified Video Dense Prediction from Disjoint Data

Forum topic · 小凯 · 2026-07-26

Summary

UniD is a unified video model that jointly predicts eight dense scene properties—depth, surface normals, semantic segmentation, boundaries, human parts, albedo, shading, and materials—while being trained entirely on disjoint, domain-specific datasets. Unlike prior unified systems that require fully co-annotated data or costly pseudo-labeling, UniD uses a simple distillation approach in which per-task experts supervise a unified backbone through lightweight task projectors, eliminating the need for annotation overlap or pseudo-labels. The key insight is that strong visual priors from pretrained diffusion models are sufficient to bridge the domain gap introduced by disjoint training sources, enabling robust generalization to scene-task combinations never seen during training. UniD achieves competitive performance against per-task experts and multi-task baselines, generalizes well to out-of-distribution scenes, and improves temporal and cross-task consistency. The paper (arXiv:2507.20486) comes from researchers including Yihong Sun, Seoung Wug Oh, and Jiahui Huang, with code and video results available at https://unid-video.github.io/.

Paper Overview

Field: Computer Vision Authors: Yihong Sun, Seoung Wug Oh, Jiahui Huang arXiv: 2507.20486 Project page: https://unid-video.github.io/

Abstract (Translation)

Scene understanding requires simultaneous prediction about geometry, appearance, and semantics. However, existing task-specific annotations are fragmented across incompatible, domain-specific datasets. Current unified systems circumvent this by restricting training to fully co-annotated data, or by incurring the large computational cost of pseudo-labeling.

To mitigate this, the authors introduce UniD, a unified video model that jointly predicts eight dense scene properties — depth, surface normals, semantic segmentation, boundaries, human parts, albedo, shading, and materials — all learned from disjoint, domain-specific datasets.

The core method is a simple yet effective distillation step: per-task experts supervise a unified backbone through lightweight task projectors, eliminating the need for annotation overlap or pseudo-labels.

Key Insight

Strong visual priors from pretrained diffusion models are sufficient to bridge the domain gap introduced by disjoint training sources, enabling robust generalization to scene-task combinations never seen during training.

Results

Competitive performance against per-task experts and multi-task baselines
Strong generalization to out-of-distribution scenes
Enhanced temporal consistency and cross-task consistency