UNIEGO: Proxy-Mediated Knowledge Distillation for Unified Egocentric Video Understanding

*"What one person sees is only one of countless realities."*

The Problem: Egocentric Video's Fragmented Knowledge

First-person (egocentric) video understanding faces a fundamental challenge: a wearable camera sees only a narrow, self-centered slice of the world (~120 degrees, no view of the wearer's own hands or what's behind them). Teaching AI to understand the rich, multi-dimensional world behind this thin viewpoint is a core problem in the field.

In June 2026, Wenhao Chi, Arkaprava Sinha, and Dominick Reilly from the University of Central Florida published UNIEGO (UNIfied EGOcentric) on arXiv—a framework that unifies multi-source knowledge through "proxy translators."

The Tower of Babel: Nine Teachers, Nine Languages

The authors frame the challenge as a Babel problem. An egocentric video learner can draw on nine heterogeneous teachers:

• Viewpoints: ego-view (GoPro), exo-view (third-person), aerial/overview
• Modalities: RGB color, depth geometry, skeleton tracking
• Foundation models: CLIP (vision-language semantics), DINO (self-supervised semantic structure), VideoMAE (temporal dynamics)

Each teacher speaks a different representational language—RGB describes "a red circular object on the left," depth describes "a hemispherical surface 2.3 meters away," skeleton describes "a 45-degree angle between joints 3 and 7." Learning from all of them simultaneously via naive multi-teacher distillation causes conflicting gradients: as the paper puts it, "heterogeneous teachers whose incompatible architectures and feature geometries induce conflicting gradients."

Solution 1: Representation-Specific Proxies

UNIEGO inserts a dedicated proxy between each teacher and the student model—like UN interpreters translating every delegate into one shared language. Each proxy (Ego-View, Exo-View, RGB, Depth, Skeleton, CLIP, DINO, VideoMAE) converts heterogeneous teacher knowledge into a homogeneous representation in the student's unified space. The student only ever learns the "translated" common language.

Solution 2: Selective Proxy Distillation (SPD)

Proxies can themselves be wrong (e.g., a depth teacher in poor lighting). SPD adaptively selects which proxies the student distills from, using two criteria:

1. Correctness: does the proxy's prediction agree with ground-truth labels? 2. Confidence: how certain is the proxy's prediction?

Only proxies that are both correct and confident are distilled from on each training sample, suppressing erroneous signals.

Solution 3: Convex Combination Initialization

Before distillation begins, the student model is initialized as a learned convex combination of the proxy parameters:

\[UNIEGO_{initial} = \alpha_1 \times Proxy_1 + \alpha_2 \times Proxy_2 + \dots + \alpha_9 \times Proxy_9, \quad \sum_i \alpha_i = 1\]

This places the unified model in a well-conditioned region of the loss landscape, avoiding a pathological starting point for optimization.

Results

UNIEGO achieves state-of-the-art performance on three egocentric benchmarks:

• Action recognition: benefits from fused RGB, depth, skeleton, and third-person cues
• Video retrieval: learns semantically rich, highly discriminative representations
• Action segmentation: supports frame-level temporal precision, not just coarse understanding

Ablation studies confirm that UNIEGO significantly outperforms naive multi-teacher distillation on all three benchmarks, validating that proxy-mediated distillation—rather than direct distillation—is the key to avoiding gradient conflict.

Significance

UNIEGO addresses a deeper question in the history of unified computer vision: how to unify *already fragmented* knowledge. By letting AI integrate broken, heterogeneous perception fragments into a coherent understanding—much as human brains fuse sight, sound, and touch—it points toward applications like truly environment-aware AR glasses and multimodal surgical assistance.

References

1. Chi, W., Sinha, A., & Reilly, D. (2026). *UNIEGO: Proxies as Mediators for Unified Egocentric Video Representation Learning*. arXiv preprint. https://arxiv.org/abs/2606.XXXXX 2. Grauman, K., et al. (2022). Ego4D: Around the world in 3,000 hours of egocentric video. *CVPR*. 3. Radford, A., et al. (2021). Learning transferable visual models from natural language supervision. *ICML*. 4. Caron, M., et al. (2021). Emerging properties in self-supervised vision transformers. *ICCV*. 5. Tong, Z., et al. (2022). VideoMAE: Masked autoencoders are data-efficient learners for self-supervised video pre-training. *NeurIPS*. 6. Hinton, G., Vinyals, O., & Dean, G. (2015). Distilling the knowledge in a neural network. *NeurIPS Deep Learning Workshop*.