What 30,000 Hours of Ego-centric Video Does Not Teach World Models

Forum topic · 小凯 · 2026-10-10

Summary

This arXiv paper (2610.12464) investigates how far scaling ego-centric human video can advance world models toward practical deployment. The authors train on a dataset of 30,000 hours of video spanning over 1,000 scene types and 14,000 contributors, and evaluate agent modeling and object-interaction fidelity directly on a challenging out-of-distribution benchmark rather than opaque downstream metrics. Scaling training data by 100x improves both capabilities but unevenly: agent modeling becomes strong, while object fidelity remains far lower and improves slowly. Notably, agent gains need not come from data—a careful visual conditioning design saturates agent fidelity with only a fraction of the data, allowing the team to isolate and measure object fidelity and find its saturation point. A new supervision scheme that shifts model capacity from scene appearance to object dynamics improves object fidelity, though a substantial gap remains. Findings transfer to downstream humanoid robot modeling. The conclusion: scaled ego-centric data brings agent modeling near its limit, but understanding of the physical world lags far behind—closing this gap depends on how models are trained, not just how much data they see.

Paper Overview

Field: Computer Vision Authors: Jiahua Dong, Anurag Bagchi, Yash Jangir, Muhammad Zubair Irshad, Sergey Zakharov, Martial Hebert, Homanga Bharadhwaj, Yu-Xiong Wang, Vitor Campagnolo Guizilini, Pavel Tokmakov Published: 2026-10-08 arXiv: 2610.12464

Original Abstract

World models offer a promising alternative to physics-based simulators, yet remain far from practical deployment. We ask how far scaling ego-centric human video takes them, using a dataset of 30,000 hours spanning over 1,000 scene types and 14,000 contributors. Rather than relying on opaque downstream metrics, we directly evaluate agent and object-interaction fidelity on a challenging out-of-distribution benchmark. Increasing training data by 100x improves both, but unevenly: the agent is modeled well, while object fidelity remains far lower and improves slowly. We show that the agent gains need not come from data, and a careful visual conditioning design saturates fidelity with a fraction of it, which lets us measure object fidelity on its own and discover its saturation point. We then introduce a supervision scheme that reallocates model capacity from scene appearance to object dynamics, improving object fidelity, though a substantial gap remains. Conclusions transfer to downstream humanoid robot modeling. Overall, scaled ego-centric data brings agent modeling near its limit, while understanding of the world lags far behind; closing this gap depends on how models are trained, not just how much data they see.

Key Takeaways

100x more ego-centric training data improves agent modeling and object fidelity, but unevenly — object fidelity stays low and improves slowly.
Agent modeling can be saturated with a small fraction of the data through careful visual conditioning design.
Isolating object fidelity reveals its own saturation point, suggesting data scale alone cannot fix it.
A supervision scheme redirecting model capacity from scene appearance to object dynamics improves object fidelity.
Findings are validated on downstream humanoid robot modeling tasks.

--- *Auto-collected on 2026-10-10*