UniEgo: Agent-Mediated Unified First-Person (Egocentric) Video Representation Learning

Forum topic · 小凯 · 2026-06-23

Summary

UniEgo is a unified egocentric video encoder developed through a hierarchical multi-teacher distillation framework, presented in the arXiv paper 2506.18497 (June 2025) by Wenhao Chi, Arkaprava Sinha, and Dominick Reilly. Egocentric video understanding is constrained by the narrow perspective of wearable cameras, so a single viewpoint, modality, or model cannot fully capture human action. UniEgo addresses this by training with nine teachers spanning first- and third-person viewpoints, RGB, depth, and skeleton modalities, and four foundation models. Instead of distilling directly from heterogeneous teachers with incompatible architectures and feature geometries, the framework inserts representation-specific agent models that convert diverse teacher knowledge into a homogeneous egocentric space. A second stage, Selective Proxy Distillation (SPD), adaptively selects a subset of proxies that are both correct and confident for each training sample, distilling only from reliable supervision and suppressing erroneous signals. UniEgo achieves state-of-the-art performance on action recognition, video retrieval, and action segmentation across three challenging ego-exo benchmarks.

Paper Overview

Field: Computer Vision (CV) Authors: Wenhao Chi, Arkaprava Sinha, Dominick Reilly Published: 2025-06-23 arXiv: 2506.18497

Abstract

Egocentric video understanding is inherently limited by the narrow perspective of wearable cameras: a single viewpoint, a single modality, a single model cannot capture the full richness of human action. The authors argue that a truly expressive egocentric representation must subsume complementary knowledge across viewpoints, modalities, and foundation model representations, yet remain deployable from egocentric video alone.

To this end, the paper introduces a hierarchical multi-teacher distillation framework that produces UniEgo, a unified egocentric encoder trained with nine teachers spanning:

First-person (ego) and third-person (exo) viewpoints
RGB, depth, and skeleton modalities
Four foundation models

Rather than distilling directly from heterogeneous teachers whose incompatible architectures and feature geometries induce conflicting signals, the framework inserts a layer of representation-specific agent (proxy) models that transform diverse teacher knowledge into a homogeneous egocentric space.

In a second stage, Selective Proxy Distillation (SPD) adaptively selects, for each training sample, a subset of proxies that are both correct and confident — distilling only from reliable supervision and suppressing erroneous signals.

Results

UniEgo achieves state-of-the-art performance on action recognition, video retrieval, and action segmentation tasks across three challenging ego-exo benchmarks.

---

*Auto-collected on 2026-06-23*