UNIEGO: Proxies as Mediators for Unified Egocentric Video Representations

Forum topic · 小凯 · 2026-06-20

Summary

UNIEGO is a unified egocentric video encoder introduced in arXiv paper 2506.16806 by Wenhao Chi, Arkaprava Sinha, and Dominick Reilly. Egocentric video understanding is limited by the narrow perspective of wearable cameras, where a single viewpoint, modality, or model cannot fully capture human action. The authors propose a hierarchical multi-teacher distillation framework that trains a single encoder with nine teachers spanning ego-exocentric viewpoints, RGB, depth, and skeleton modalities, plus four foundation models. Instead of distilling directly from heterogeneous teachers whose incompatible architectures cause gradient conflicts, the framework inserts modality-specific proxy models that translate diverse teacher knowledge into a homogeneous egocentric space. A second stage, Selective Proxy Distillation (SPD), adaptively selects correct and confident proxy subsets per sample to suppress erroneous supervision, and is stabilized by initializing UNIEGO as a learned convex combination of proxy parameters. UNIEGO achieves state-of-the-art results on three egocentric tasks—action recognition, video retrieval, and action segmentation—across three ego-exo benchmarks, outperforming naive multi-teacher distillation baselines.

Paper Overview

Field: Computer Vision
Authors: Wenhao Chi, Arkaprava Sinha, Dominick Reilly
Published: 2025-06-20
arXiv: 2506.16806

Abstract

Egocentric video understanding is inherently limited by the narrow perspective of wearable cameras: a single viewpoint, a single modality, a single model cannot capture the full richness of human action. We argue that a truly expressive egocentric representation must subsume complementary knowledge across viewpoints, modalities, and foundation model representations, yet remain deployable from egocentric video alone.

To this end, the authors introduce a hierarchical multi-teacher distillation framework that produces UNIEGO, a unified egocentric encoder trained with nine teachers spanning ego-exo viewpoints, RGB, depth, and skeleton modalities, and four foundation models.

Method Highlights

Rather than distilling directly from heterogeneous teachers—whose incompatible architectures and feature geometries induce gradient conflicts—the framework places representation-specific proxy models as an intermediate layer, translating diverse teacher knowledge into a homogeneous egocentric space.
The second distillation stage, Selective Proxy Distillation (SPD), adaptively selects a subset of correct and confident proxies for each training sample, letting only reliable supervisors guide the distillation and suppressing erroneous signals.
SPD is further stabilized by initializing UNIEGO as a learned convex combination of proxy parameters, placing the unified model in a well-conditioned region of the loss landscape before distillation begins.

Results

UNIEGO achieves state-of-the-art performance on three egocentric video understanding tasks (action recognition, video retrieval, and action segmentation) across three challenging ego-exo benchmarks, outperforming naive multi-teacher distillation baselines. This confirms that structured, proxy-mediated knowledge transfer produces richer and more discriminative egocentric representations.

*Source: arXiv:2506.16806*