MILO: Reconstructing Humans and Objects in Interaction Using Large Reconstruction Models

Forum topic · 小凯 · 2026-08-30

Summary

MILO is a framework for 3D human-object interaction (HOI) estimation from a single image, presented by Agniv Chatterjee and Georgios Pavlakos (arXiv:2608.27407). Unlike prior methods that rely on reprojection and contact constraints to fit parametric human models and object templates to 2D images, MILO leverages Large Reconstruction Models (LRMs) as a geometric scaffold. The key insight is that LRMs preserve relative human-object arrangement and proximity cues, which simplifies reconstruction. The pipeline interprets the LRM-generated mesh: it is segmented into human and object components, a parametric body model is fitted to the human part, and object templates are aligned to the object part when available. MILO achieves strong reconstruction accuracy across multiple benchmarks and interaction scenarios, outperforming existing baselines. The work has applications in AR/VR, robotics, and embodied AI, addressing challenges such as depth ambiguity, occlusion, and object shape variability.

Paper Overview

Field: Computer Vision
Authors: Agniv Chatterjee, Georgios Pavlakos
Published: 2026-08-27
arXiv: 2608.27407

Abstract (Original)

Estimation of Human-Object Interactions in 3D (3D HOI) is a fundamental problem in 3D computer vision with applications in AR/VR, robotics, and embodied AI. However, reconstructing these interactions in 3D remains challenging due to depth ambiguities, occlusions, and object shape variability. Existing approaches are primarily concerned with reprojection and contact constraints, fitting parametric human models and object templates to 2D images. In this paper, we explore a different avenue. We present MILO, a framework that leverages the visual capabilities of Large Reconstruction Models (LRMs) to recover detailed 3D human-object interactions from a single image. Our key observation is that LRMs provide a powerful geometric scaffold that preserves relative human-object arrangement and proximity cues.

Summary

3D human-object interaction (3D HOI) estimation is a foundational problem in 3D computer vision, with applications in AR/VR, robotics, and embodied AI. Reconstructing such interactions in 3D remains difficult due to depth ambiguity, occlusion, and object shape variability. Most existing methods rely on reprojection and contact constraints, fitting parametric human models and object templates to 2D images.

This paper takes a different approach. The authors propose MILO, a framework that uses the visual capabilities of Large Reconstruction Models (LRMs) to recover detailed 3D human-object interactions from a single image. The key observation is that LRMs provide a strong geometric scaffold that preserves relative human-object arrangement and proximity cues. This significantly simplifies reconstruction by re-framing the problem as interpreting the LRM mesh:

1. Segment the LRM mesh into human and object components. 2. Fit a parametric body model to the human component. 3. Align object templates to the object component (when templates are available).

MILO achieves strong reconstruction accuracy across multiple benchmarks and interaction scenarios, outperforming existing baselines.

---

*Auto-collected on 2026-08-30 from zhichai.net.*

MILO: Reconstructing Humans and Objects in Interaction Using Large Reconstruction Models

Summary

Paper Overview

Abstract (Original)

Summary

Tags