Summary
EgoGeneralist is a framework that adapts pretrained video generators to unify action-conditioned egocentric video generation, pose estimation, and joint action-video generation. Modern video models have strong visual priors but do not natively represent human action in metric 3D space. The authors introduce a role-adaptive action renderer providing video-native representations for multidirectional action-video learning, so metric motion serves both as a spatial control signal and as a recoverable generation target. Cross-stream action-video coupling builds tight correspondence between metric motion and visual dynamics through lightweight adaptation while preserving pretrained visual priors. Together, these components form a generalist framework covering five action-video tasks. Although adapted only on controlled laboratory recordings, EgoGeneralist generalizes to diverse unseen scenes and interactions with effective action control and visual quality, bridging pretrained visual knowledge and metric 3D human motion.
论文概要
研究领域: CV
作者: Yanlin Jin, Bowen Yang, Cheng Zhang, Yifan Yang, Zhe Li, Jianfei Cai, Kai Zhu
发布时间: 2026-11-05
arXiv: 2611.08477
Abstract
Modern video models possess strong visual priors, yet do not natively represent human action in metric 3D space. We present EgoGeneralist, which adapts pretrained video generators to unify action-conditioned egocentric video generation, pose estimation, and joint action-video generation. Our role-adaptive action renderer provides video-native representations for multidirectional action-video learning, making metric motion usable as both a spatial control signal and a recoverable generation target. Cross-stream action-video coupling establishes close correspondence between metric motion and visual dynamics through lightweight adaptation while preserving the pretrained visual prior. Together, these designs establish a generalist framework spanning five action-video tasks. Despite adaptation only on controlled laboratory recordings, EgoGeneralist generalizes to diverse unseen scenes and interactions, with effective action control and visual quality, bridging pretrained visual knowledge and metric 3D human motion.
Links
- arXiv: https://arxiv.org/abs/2611.08477
This page is an English static mirror generated for search and AI citation.
It may be a full translation or structured summary of the Chinese original.
Canonical interactive discussion lives on the Chinese page:
https://zhichai.net/topic/178636823