UNIEGO: Proxies as Mediators for Unified Egocentric Video Representation Learning

Paper Overview

Research Areas: cs.CV, cs.LG Authors: Wenhao Chi, Arkaprava Sinha, Dominick Reilly Published: 2026-06-21 arXiv: 2506.17586

Summary

Egocentric video understanding is fundamentally limited by the narrow viewpoint of wearable cameras: a single viewpoint, single modality, and single model struggle to capture the full richness of human behavior. The authors argue that a truly expressive egocentric representation must integrate complementary knowledge from different viewpoints, modalities, and foundation models, while remaining deployable from egocentric video alone.

They propose a hierarchical multi-teacher distillation framework to train UNIEGO, a unified egocentric encoder guided by nine teachers covering ego-exo viewpoints, RGB, depth, and skeleton modalities, as well as four foundation models.

Since teacher architectures and feature geometries differ, direct distillation can produce conflicting gradients. The framework therefore inserts representation-specific proxy models that first transform diverse teacher knowledge into a homogeneous egocentric space. A second stage — Selective Proxy Distillation (SPD) — adaptively selects, per training sample, a subset of proxies that are both correct and confident, distilling only from reliable supervision and suppressing erroneous signals.

To further stabilize training, UNIEGO is initialized as a learned convex combination of proxy parameters, placing the unified model in a well-conditioned region of the loss landscape before distillation begins.

UNIEGO achieves state-of-the-art performance on three egocentric video understanding tasks (action recognition, video retrieval, and action segmentation) across three challenging ego-exo benchmarks, significantly outperforming naive multi-teacher distillation baselines. This shows that structured, proxy-mediated knowledge transfer yields richer, more discriminative egocentric representations.

In Plain Terms

Imagine wearing a head-mounted camera that only sees your own hands, yet wanting to understand complex behaviors that truly require "seeing from the side" or "from a skeleton perspective."

Previous approaches pulled in multiple teachers (different viewpoints, sensors, and large models) to teach directly — but the teachers contradicted each other, confusing the student.

UNIEGO's approach is smarter:

1. First, "translators" (proxy models) convert each teacher's knowledge into the egocentric "language" the student understands. 2. During training, the student listens selectively: for each lesson, it only listens to the translators who are both correct and confident, ignoring vague or wrong ones. 3. On day one, the student starts from the "average position" of all translators' wisdom (convex-combination initialization), giving it a stable starting point.

The result: a student that has only seen first-person footage makes better judgments than many multi-teacher methods, ranking first on action recognition, video retrieval, and action segmentation — capturing the essence of the world's smartest teachers cleanly, from a single egocentric camera.

--- *Auto-collected on 2026-06-21*