MILO: Reconstructing Humans and Objects in Interaction Using Large Reconstruction Models

Forum topic · 小凯 · 2026-08-30

Summary

MILO is a new framework for 3D human-object interaction (HOI) estimation from a single image, presented by Agniv Chatterjee and Georgios Pavlakos (arXiv:2608.27407). Instead of fitting parametric human models and object templates to 2D images via reprojection and contact constraints—as most prior methods do—MILO leverages Large Reconstruction Models (LRMs) to recover detailed 3D interactions. The key insight is that LRMs act as a powerful geometric scaffold that preserves relative human-object arrangement and proximity cues despite depth ambiguity, occlusion, and object shape variability. MILO reformulates reconstruction as interpreting the LRM's predicted mesh: it segments the mesh into human and object components, fits a parametric body model to the human part, and aligns object templates to the object part when available. The authors report strong reconstruction accuracy across multiple benchmarks and interaction scenarios, outperforming existing baselines. Applications include AR/VR, robotics, and embodied AI.

Paper Overview

Field: Computer Vision (CV)
Authors: Agniv Chatterjee, Georgios Pavlakos
arXiv: 2608.27407

Abstract

Estimation of Human-Object Interactions in 3D (3D HOI) is a fundamental problem in 3D computer vision with applications in AR/VR, robotics, and embodied AI. However, reconstructing these interactions in 3D remains challenging due to depth ambiguities, occlusions, and object shape variability. Existing approaches are primarily concerned with reprojection and contact constraints, fitting parametric human models and object templates to 2D images. In this paper, we explore a different avenue.

We present MILO, a framework that leverages the visual capabilities of Large Reconstruction Models (LRMs) to recover detailed 3D human-object interactions from a single image. Our key observation is that LRMs provide a powerful geometric scaffold that preserves relative human-object arrangement and proximity cues. This significantly simplifies the reconstruction process, reformulating the problem as interpreting the LRM's mesh: MILO segments it into human and object components, fits a parametric body model to the human part, and aligns object templates to the object part when available.

MILO achieves strong reconstruction accuracy across multiple benchmarks and interaction scenarios, outperforming existing baselines.

*(Auto-collected on 2026-08-30)*

MILO: Reconstructing Humans and Objects in Interaction Using Large Reconstruction Models

Summary

Paper Overview

Abstract

Tags