UNIEGO: Proxies as Mediators for Unified Egocentric Video Representation Learning

Forum topic · 小凯 · 2026-06-21

Summary

UNIEGO is a unified egocentric video encoder built via hierarchical multi-teacher distillation, introduced in arXiv paper 2506.16620 by Wenhao Chi, Arkaprava Sinha, and Dominick Reilly. Egocentric video understanding is inherently limited by the narrow viewpoint of wearable cameras; a single view, modality, or model cannot capture the full richness of human actions. The authors propose that expressive first-person representations should integrate complementary knowledge across views (egocentric and exocentric), modalities (RGB, depth, skeleton), and foundation models. UNIEGO is trained from 9 teachers spanning these dimensions plus 4 foundation models. Instead of distilling directly from heterogeneous teachers whose incompatible architectures and feature geometries produce conflicting gradients, the framework inserts representation-specific proxy models that translate diverse teacher knowledge into a homogeneous egocentric space. A second stage, Selective Proxy Distillation (SPD), adaptively selects a subset of proxies that are both correct and confident for each training sample, distilling only reliable supervision and suppressing erroneous signals. SPD further stabilizes training by initializing UNIEGO as a learned convex combination of proxy parameters. UNIEGO achieves state-of-the-art results on action recognition, video retrieval, and action segmentation across three challenging egocentric/exocentric benchmarks, outperforming naive multi-teacher distillation baselines.

Paper Overview

Field: Computer Vision Authors: Wenhao Chi, Arkaprava Sinha, Dominick Reilly Published: 2026-06-20 arXiv: 2506.16620

Abstract

Egocentric video understanding is fundamentally constrained by the narrow field of view of wearable cameras: a single viewpoint, single modality, and single model cannot capture the full richness of human actions. This paper argues that truly expressive egocentric representations must encompass complementary knowledge across views, modalities, and foundation models, while remaining deployable using only egocentric video.

To this end, the authors introduce a hierarchical multi-teacher distillation framework that trains UNIEGO, a unified egocentric encoder trained from 9 teachers covering egocentric/exocentric viewpoints, RGB, depth, and skeleton modalities, as well as 4 foundation models.

Key Contributions

Proxy-mediated distillation: Rather than distilling directly from heterogeneous teachers — whose incompatible architectures and feature geometries produce conflicting gradients — the framework inserts a layer of representation-specific proxy models that translate diverse teacher knowledge into a homogeneous egocentric space.
Selective Proxy Distillation (SPD): In a second stage, SPD adaptively selects a subset of proxies that are both correct and confident for each training sample, distilling only reliable supervision and suppressing erroneous signals.
Stable initialization: SPD further stabilizes training by initializing UNIEGO as a learned convex combination of proxy parameters, placing the unified model in a well-conditioned loss region before distillation begins.

Results

UNIEGO achieves state-of-the-art performance on three egocentric video understanding tasks — action recognition, video retrieval, and action segmentation — across three challenging egocentric/exocentric benchmarks, outperforming naive multi-teacher distillation baselines. This demonstrates that structured, proxy-mediated knowledge transfer yields richer and more discriminative egocentric representations.

--- *Auto-collected on 2026-06-21*