IMU-to-4D: Seeing Without Eyes — 4D Human-Scene Understanding from Wearable IMUs

Forum topic · 小凯 · 2026-04-25

Summary

This post introduces IMU-to-4D, a research paper (arXiv:2604.21934) by Hao-Yu Hsu, Tianhang Cheng, and Jing Wen that explores vision-free 4D perception. The framework reconstructs detailed 4D human motion and coarse 3D scene structure using only data from a few everyday inertial measurement units (IMUs), such as those found in earbuds, watches, and smartphones. The key innovation is repurposing large language models for non-visual spatiotemporal understanding of human-scene dynamics, addressing persistent challenges of camera-based perception including privacy, safety, energy efficiency, and scalability. Experiments across diverse human-scene datasets show that IMU-to-4D produces more coherent and temporally stable results than state-of-the-art cascaded pipelines, demonstrating that wearable motion sensors alone can support rich 4D human-scene understanding.

Overview

Field: Computer Vision Authors: Hao-Yu Hsu, Tianhang Cheng, Jing Wen Published: 2026-04-23 arXiv: 2604.21934

Abstract

Understanding human activities and their surrounding environments typically relies on visual perception, yet cameras pose persistent challenges in privacy, safety, energy efficiency, and scalability. This work explores an alternative: 4D perception without vision — reconstructing human motion and 3D scene layouts purely from everyday wearable sensors.

Key Contribution: IMU-to-4D

IMU-to-4D is a framework that repurposes large language models (LLMs) for non-visual spatiotemporal understanding of human-scene dynamics.
It uses data from only a few inertial sensors found in earbuds, watches, or smartphones.
The model predicts detailed 4D human motion together with coarse scene structure.

Results

Experiments across diverse human-scene datasets show that IMU-to-4D yields more coherent and temporally stable results than state-of-the-art (SoTA) cascaded pipelines, indicating that wearable motion sensors alone can support rich 4D human-scene understanding.

---

*Auto-collected on 2026-04-25*