UNIEGO: Proxies as Mediators for Unified Egocentric Video Understanding
> *"What one person sees is only one of a million truths."*
The Problem: A Tower of Babel for Egocentric Video
A GoPro-style first-person camera captures a narrow, self-centered slice of the world—roughly 120 degrees, no hands, no behind-the-back context. The core challenge of egocentric video understanding is: how can AI infer the rich, multi-dimensional world behind this impoverished viewpoint?
The surprising answer proposed by UNIEGO (UNIfied EGOcentric), a 2026 arXiv paper by Wenhao Chi, Arkaprava Sinha, and Dominick Reilly of the University of Central Florida, is not more cameras—it's having the model "listen to nine teachers" who each speak a different language:
- Views: ego-centric (head-mounted), exocentric (third-person), and aerial/top-down
- Sensors: RGB, depth, skeleton pose
- Foundation models: CLIP (vision-language semantics), DINO (self-supervised semantic structure), VideoMAE (temporal dynamics)
- Ego-View Proxy, Exo-View Proxy
- RGB Proxy, Depth Proxy, Skeleton Proxy
- CLIP Proxy, DINO Proxy, VideoMAE Proxy
Each teacher is valuable, but their outputs are mutually incompatible—one describes a red object on the left, another a hemispherical surface 2.3 meters away, a third a 45-degree joint angle. The paper calls this *"heterogeneous teachers whose incompatible architectures and feature geometries induce conflicting gradients."*
Naive multi-teacher distillation—training the student on all teachers simultaneously—produces conflicting, cancelling gradients: a modern Tower of Babel.
Solution 1: Proxies as Simultaneous Interpreters
UNIEGO never lets the student hear the raw teachers. Instead, it inserts a representation-specific proxy between each teacher and the student, like UN simultaneous interpreters translating every delegate into one shared language:
Solution 2: Selective Proxy Distillation (SPD)
Proxies can be wrong—for example, a Depth proxy in a dark scene may hallucinate. SPD adaptively selects, per training sample, which proxies to listen to, using two criteria:
1. Correctness — does the proxy's prediction agree with the ground truth (e.g., the labeled action "chopping vegetables")? 2. Confidence — how certain is the proxy about its prediction?
Only proxies that are both correct and confident are distilled from; unreliable signals are suppressed. It's like an editor telling a reporter which witnesses to trust.
Solution 3: Convex-Combination Initialization
Training is like a blind climber descending a loss landscape: start at a cliff edge and gradients explode; start in a swamp and you get stuck in local minima. UNIEGO initializes the student as a learned convex combination of the proxy parameters:
with the weights summing to 1. This places the unified model in a well-conditioned region of the loss landscape before distillation begins—a good starting map for the climber.
Results: SOTA Across Three Tasks
UNIEGO achieves state-of-the-art performance on:
1. Egocentric action recognition — recognizing "chopping" from head-mounted video; benefits from fused RGB, depth, skeleton, and third-person cues. 2. Video retrieval — finding the exact clip matching a text query (e.g., "me frying an egg in the kitchen"), requiring semantically rich yet discriminative representations. 3. Action segmentation — frame-level labeling of long videos ("0:00–2:30 washing, 2:30–5:00 chopping..."), the hardest task, showing proxy-mediated transfer supports fine-grained parsing.
Ablation studies confirm the core hypothesis: direct distillation fails; proxy-mediated distillation works. Direct distillation tears the student in nine directions, yielding a poor compromise; proxies divide-and-conquer the conflict, and SPD filters out bad translations.
Why It Matters
Applications include AR smart glasses that genuinely understand the scene ("you're washing vegetables; tomatoes and eggs are on the counter") and surgical assistance fusing tissue appearance, instrument depth, and overview camera views.
UNIEGO extends computer vision's long tradition of unification—unified feature learning, unified architectures, unified multimodal perception—by tackling a deeper problem: unifying what is already fragmented. Like Borges' Aleph, it points toward reaching comprehensive, unified understanding from a limited, partial, first-person viewpoint.
References
1. Chi, W., Sinha, A., & Reilly, D. (2026). *UNIEGO: Proxies as Mediators for Unified Egocentric Video Representation Learning*. arXiv preprint. https://arxiv.org/abs/2606.XXXXX 2. Grauman, K., et al. (2022). Ego4D: Around the world in 3,000 hours of egocentric video. *CVPR*. 3. Radford, A., et al. (2021). Learning transferable visual models from natural language supervision. *ICML*. 4. Caron, M., et al. (2021). Emerging properties in self-supervised vision transformers. *ICCV*. 5. Tong, Z., et al. (2022). VideoMAE: Masked autoencoders are data-efficient learners for self-supervised video pre-training. *NeurIPS*. 6. Hinton, G., Vinyals, O., & Dean, J. (2015). Distilling the knowledge in a neural network. *NeurIPS Deep Learning Workshop*.